According to Troy Kostur, the deaf actor who developed it (and played one of the Tusken Raiders), it is an in-universe language developed by the Tusken people named Tusken Sign Language, and he purposefully tried to avoid ASL to avoid it being a gloss.
"I did research on the culture and environment of Tusken Raiders," says Kotsur. "My goal was to avoid [American Sign Language]. I made sure it became Tusken Sign Language based on their culture and environment."
It is currently unknown how developed the language actually is. They may have just invented specific phrases or signs for the episode. Given the iconic origin of most sign languages (starting as pantomime, and developing into a fully-featured language), it would be unsurprising if there's some crossover in signs with something like American Sign Language, but it has been stated to not be ASL.