According to this article on Wookieepedia:
The character Nien Nunb was realized as a puppet for his appearance in
Return of the Jedi. He was performed by Mike Quinn and Richard
Bonehill. Mike performed the acting in the cockpit, which was a large
animatronic hand puppet and Richard was the extra who was in the full
body suit.
See also this interview with Mike Quinn.
They had shot some of the background walk-around Nien scenes with the extra, Richard Bonehill, already by this time.
...
I was looking at one of the masks and was fiddling with it. Then, Phil
said that some scenes were coming up with this guy in the Falcon and
now he had to speak dialogue. The problem was that he hadn't been
designed and built for that. They were thinking they could put air
bladders in his cheeks for movement, but his mouth and eyes were
static and the only way around it was to cover his mouth with one of
those oxygen masks. That's why there are a few pictures that have
this. Also, you'll see the Nien scenes in the briefing room he has
this hanging from his neck. I put my hand inside the mouth and began
moving it like a hand puppet. Phil liked what he saw and suggested it
to George.