Leia is speaking Ubese
Boushh is an Ubese and he (she when it is Leia disguised as Boushh) speaks Ubese
This is shown in the Return of the Jedi novelization by James Kahn:
"He bowed low and then spoke in Ubese.
"Greetings, Majestic One. I am Boushh." It was a metallic language, well-adapted to the rarefied atmosphere of the home planet from which this nomadic species arose.
Jabba spoke in the same tongue, although his Ubese was stilted and slow.
And in the Return of the Jedi film.
Although the language was not named on screen, it was identified as such in the subtitles of the 2006 Limited Edition version on both re-mastered and original cuts, which was released on 12th of September 2006.
"I have come for the bounty on this Wookiee." - Boushh speaking Ubese
The script for the film also confirms this:
BOUSHH (In Ubese subtitled)
I have come for the bounty on this Wookiee
To get further details of the language such as why they use an amplifier we have to go into Legends and the game Star Wars: Galactic Backgrounds:
Ubese vocal cords could not produce sound above raspy whispers, making
them difficult to understand when their voices weren't amplified.
The Ubese also developed a sign language, Ubeninal, as an alternative to
communicate with other Ubese. Ubeninal had no written form.
Most Ubese were fluent in Ubese language, Ubeninal and Basic, the common language of the galaxy, although they were only literate in their own language.
Ubese rarely chose to speak Basic.