There seem to be many races who have telepathic abilities in the Star Trek universe. Is there any scientific explanation mentioned in any of the series that explain how it's possible?
ST:Voyager episode/s established the existence of Psionic energy and a Psionic field permeating through space. I believe psionic energy has been mentioned in TOS episodes too, but I can't recall exact episodes.
Quote from Memory-Alpha Telepathy article (which itself quotes now defunct star-trek-voyager.net):
Star Trek science consultant and writer André Bormanis has revealed that telepathy within the Star Trek universe works via the "psionic field." According to Bormanis, a psionic field is the "medium" through which unspoken thoughts and feelings are communicated through space. Some humanoids can tap into this field through a kind of sense organ located in the brain (e.g. the paracortex). In the same manner that Human eyes can sense portions of the electromagnetic field, telepaths can sense portions of the psionic field
Although, note that this is an in-universe explanation rather than a scientific explanation. Saying that telepathy arises from a 'psionic field' is really a case of naming-the-problem-as-the-solution.
In Where No Man has Gone Before (TOS), Elizabeth Dehner says, "ESPers are simply people with flashes of insight." They have no idea what causes it. But then, in Plato's Stepchildren (TNG), they find that kironide, which is in the local foods, is what causes their telekinetic ability.
Troi never attempts to describe her abilities, but it is blocked in The Loss by the two dimensional beings.
It may be of some significance that psi abilities are not dealt with at all the the Star Trek: The Next Generation Writer's Technical Manual (At least not as of the 4th season). In the Star Trek: The Next Generation Writers'/Directors' Guide (in other words the series bible) has a section titled The Story -- What Doesn't Work and this is the 2nd subsection:
2. We do not do stories about psi-forces or mysterious psychic powers
No matter how fantastic the events in a story, the explanation must be extrapolated from a generally accepted science theory. (We have accepted the telepathy of Counselor Deanna Troi because many reputable scientists have acknowledged the possibility of such abilities, but you will note that we have limited Troi to "reading" only emotions.)
So they do not provide any guidance to the writers on how psi powers work, even though they have whole sections in the tech guide on things like phasers, transporters, and even smaller sections on artificial gravity. This gives a strong indication that the producers, at least of ST:TNG, did not want to get into the causes or workings of psionics.
While I'll link to the article on telepath at Memory Alpha, I won't go into it since it's already been quoted here. But I think it's interesting to note that, while the later series like TNG and VOY used technobabble to explain details of almost every system on a starship, very little was ever said online about psi abilities.
Star Trek explains this in different ways, not always consistently (there's a lot of Star Trek -- that's going to happen). In the TNG episodes "Dark Paige" and "Eye of the Beholer," psilosynine was a neurotransmitter and one of the main chemicals involved in Betazoid telepathy. But in "Haven" Lwaxana Troi told Wyatt (Troi's fiance) that he'd been seeing visions of his eventual mate (not Troi -- it's complicated) because "everything is connected."
Which didn't explain why Wyatt didn't see Picard, Mr. Mott, or B'awhb, the guy who picks up after tribbles on Epsilon Draconis forty-seven.