In episode 1x05 The Last Outpost the Ferengi antagonists are thought to be a formidable enemy. The crew mentions, without hesitation, the technical formidability of the Ferengi and it is assumed that it is technically feasible for a Ferengi ship to disable a galaxy class starship. In contrast, in Deep Space nine the Ferengi, Quark and Rom especially, are more like the comic relief, and the Ferengi tech we see is much less threatening and not at all combat oriented. When did the Ferengi stop being a formidable enemy, whether it was a change in "tune" of script writing or a specific event in the series?
It was essentially with the episode you mention, The Last Outpost. I heard David Gerrold talk at a convention toward the end of or after the end of the first season of Star Trek: The Next Generation. He said they wanted a small, compact, bobcat like creature - one that could tear you to pieces.
They were disappointed because what they got on screen was nothing like what they had intended. They didn't come across seriously enough to project the feeling of a threat, so they were dropped as serious antagonists at that point.
While they did show up during that season in other roles, note that it wasn't too far into the season (for instance, The Stargazer). While the disappointment hit them when they saw the episode on screen, the series had a long lead time from start to finish for each episode, so it wasn't possible to adapt immediately to the situation.
Short Answer: They are not less formidable. It was later revealed their ships were individually powerful but not as numerous as originally implied.
Detailed Answer: Referenced in Wikipedia:
The Ferengi warship shown in the Last Outpost was a D'Kora-class warship. While it is mainly used as a tradeship, the D'Kora-class is capable in battle, carrying photon torpedoes and disruptors. Tactically it is about as powerful as a Galaxy-class starship, but only has half as many crew.
The Ferengi have ships that are powerful but they are not interested in fighting potential clients. Their Laws of Acquisition promote the idea of creating and finding customers (marks) wherever they can. The Ferengi Alliance is economically powerful but despite the capabilities of their ships, they are not inclined toward conquest, though pirates are known to exist among the Ferengi who are inclined to attack ships, steal their cargo and sell it for a profit. Their military is rather small in comparison to other Alpha Quadrant races.
The Ferengi are mercantile and would rather buy technology rather than create it. So while they have technology on par with the Federation, most of it is acquired through commerce, not innovation. They seem to be able to adapt technologies well enough, acquiring their first warp drive from the Breen.
The Ferengi are rather unique in the Federation in that they have a history that does not include any major wars or genocides. The Ferengi Alliance was one of the few races that did not participate in the Dominion War, suffering no losses other than economic ones.
Note how small the Ferengi Alliance is on this map of the Federation: http://cdn.booredatwork.com/wp-content/uploads/2009/10/Star_trek_map2.jpg
I'm not sure if there is an in-universe answer for this. The truth is that they were meant to be formidable originally, but audiences didn't react to them in the way the producers had hoped. This is covered in several interviews and articles, including this one.
Quote from the article:
There was a time when the Ferengi were meant to be mean and vicious, not greedy and silly.
Armin Shimerman originally played a Ferengi in the first season of Star Trek: The Next Generation: The Last Outpost where the Ferengi were supposed to be very different than what they ended up being on The Next Generation and subsequent Star Trek series. “The producers were very specific about what they wanted,” he explained. “There was no mention of comedy whatsoever, rather, they said that they were like old turn-of-the-century Chinese clipper captains. They were vicious competitors, and were even capable of eating their enemies. They were envisioned as the new Klingons and of course it never happened that way. They were supposed to be evil, not comic people.”
Clearly, the Federation database had some old American intelligence from 1947 that suggested the Ferengi had massively superior technology including far advanced weaponry, and possibly designs on conquering the earth.(DS9-0409 Little Green Men).
This intelligence carried through to the next encounter with them, 400 years later, where their legend had grown, perhaps, as a mysterious and incredibly advanced race.
These early records might even suggested they had mind control.
That intelligence could have been borne out "later" in TNG (The Battle)..
Not explicitly laid out in-universe, but plausible.
Basically, the ferengi might have been threatening as ridiculous 1960's scifi villains, but they just didn't work by the time of TNG.