This just rang a bell, and I found the answer. The short story is "Try to Remember" in the short story collection Eye by Frank Herbert.
The aliens arrive near Portland Oregon, destroy a Pacific Atoll and make this announcement:
Its five green-skinned, froglike occupants had delivered the ultimatum, one copy printed on velvety paper to each major government, each copy couched faultlessly in the appropriate native tongue:
"You are requested to assemble your most gifted experts in human communication. We are about to submit a problem. We will open five identical rooms of our vessel to you. One of us will be available in each room.
"Your problem. To communicate with us.
"If you succeed, your rewards will be great.
"If you fail, that will result in destruction for all sentient life on your planet.
"We announce this threat with the deepest regret. [...]"
That last line is one of my favorites of Herbert's writing.
The alien's language made deception and miscommunication of emotional content impossible, even when talking to oneself. It wasn't so much a complex language, as a call back to non-verbal communication like ancient tribal dances. (which Herbert unfortunately referred to as "primitive" and "non-rational", but he wasn't an anthropologist)
The frog aliens created the language to atone for the terrible crimes of war and violence their species perpetrated on the Galaxy.
The story can be found and read on Google Books.