[This paragraph is all TVTropes links] So, put simply and out-of-universe-ly, this is just a kind of Applied Phlebotinum called Translator Microbes (from Farscape), which is also known as Universal Translator (link redirects to Translator Microbes). The name should give a clue: It's one of few common ways of dealing with alien languages, given solely for the benefit of the viewer, and it permeates both shows.
That said, for in-universe, Star Trek has changed the explanation over the years. The first explanation, from The Original Series, was the brainwave-scanning one you mention in the question. ThinkingStiff shows quite a few issues with that explanation here.
The other explanation is that most languages follow one of several grammatical patterns, which make it easy for the universal translator to match it to one of them. Or if there are issues, to modify one of those patterns over time and come up with a new translation matrix that works for that new language. (No, this does not explain specific nouns/etc when it comes to new languages)
There are two instances of this failing that immediately come to mind:
- The infamous Darmok episode of TNG, where the Universal Translator was working perfectly but didn't have the context for a semantic translation. Their language was made up entirely of memes specific to their culture, so while we could hear the translation of each word or sentence, we didn't know what they were referencing, so the meaning didn't come across.
- The Skrreean language from DS9 was another different enough that the Universal Translator had difficulties with, but over time it was able to adjust and learn that language as well.
I can't find it at the moment, but there was an explanation in one of the ENT episodes (I believe during Season 1), where Hoshi Sato explains that most languages, alien or otherwise, are based on a limited number of patterns, which the translator is able to use as a baseline to establish a translation for. The UT's failure with some languages would simply mean that they fall outside one of these common patterns.
Now, as for why these patterns even exist? 4.5 billion years ago, the ancient humanoids seeded life across this part of the galaxy:
Believing that the life span of a single species was finite, the ancient humanoids seeded the primordial environments of many planets with a DNA code that would direct the evolution of life on that planet towards a form similar to their own.
And additionally, Sargon claimed that his people did the same thing, around 600,000 years ago.
The universal translator worked on both of these species, implying that their seeding life did not just push physical form, but also affected mental processes, causing a limited number of language patterns to form in humanoids.
While the ancient humanoids only seeded our area of the galaxy, Sargon claimed their people spread across the whole galaxy. Given the prevalence of humanoids over the entire galaxy, it's not unreasonable to assume that they (or other humanoids following in the footsteps of the ancient humanoids) spread the ancient humanoid "seed codes" for directed evolution even further than the ancient humanoids could. And therefore spreading those common language patterns through most of the rest of the galaxy's humanoids.
Despite all of this explanation, because the UT is primarily for the viewers, some inconsistencies will arise that only make sense with the brainwave-scanning explanation, and other inconsistencies will arise that cannot be explained by either explanation - for example, translations of specific nouns. A missing combadge, as mentioned in the question, is either just plain a mistake (it's where the UT is located), or the alien has their own UT. The Ferengi embed them inside their ears, for example.