The human visitor can't appreciate their music and takes a drug which allows him to hear as the aliens can hear. The drug opens his appreciation to the beauty of the alien's music, but the drug wears off, having burned out the part of his brain that allowed the drug to work, leaving him crushed with the realization of what he is now missing.
It's The Secret Sense by Isaac Asimov. I read it in the anthology The Early Asimov.
Lincoln Fields, a rich Earthling from New York living on Mars, is discussing the merits of the highly developed sense of sight and hearing that humans possess, which the Martians do not. Garth Jan, his Martian friend, counters with the merits of the senses that the Martians possess, and unintentionally lets slip that there is also one secret sense. Fields asks him to give it to him but Garth Jan refuses at that time. Six months later, using Martian social law in an unethical fashion, Fields forces him to let him feel this sense and Jan reluctantly agrees.
Fields is injected by a Martian physician with a hormone extract that will activate the sense in him, but only for five minutes, after which it will be unusable ever again. Jan's music teacher commences playing a musical composition on a Martian instrument, the portwem.