Hmm, this went a little long. Oh, well, I've already written it.
Science Fiction doesn't tell us much about future, but it tells us a lot about the time a work was written in because science fiction writers extrapolate their current knowledge and expectations to the future. You have to understand the zeitgeist of both the writers and the audience in historical context to under a work of science fiction.
In a very important way, science fiction as a genera is more important because of what it tells of the past, of history, than what it might tell us about the future.
In the case of Star Trek, the overall tone of "idealistic" vs "realistic", optimistic vs pessimistic, and even light vs dark, tracks the overall cultures attitudes towards Democratic Socialism in America.
Socialism is at it's heart an optimistic vision wherein intellectual elites can prefect humanity with the power of the state. Roddenbury' spent his life immersed in a vision of the power of American style Democratic Socialism to perfect humanity and society. It follows that he would extrapolate a future in which Democratic Socialism had so perfected humanity that individuals had to hie off to "The Final Frontier" just to find some conflict and adventure. When socialism eventually fell in the 90s, so did the optimism themes of Star Trek that were pegged to it.
Roddenbury was a member of the Greatest Generation, came of age during The Great Depression, flew as fighter pilot in WWII, etc and like most of that generation saw big government as trustworthy and effective mechanism for improving the world. After all, they had believed the US Federal government had mitigated the Great Depression, whipped the Nazis and held the Commies at bay.
The early 1960s where the high water mark of American Democratic Socialism and the Space program was it's symbolic heart. In Star Trek TOS, Roddenbury extrapolated the positive vision of Kennedy's New Frontier America two hundred years into the future. Star Fleet was just 1962's "Failure is not an option", NASA gone interstellar.
If he had made it into production five years earlier, when Kennedy was still alive and space fever was at its height, who knows how big the show would have gotten. Instead Star Trek's arrived at the wrong time and it was that very it turned out optimistic tone and confidence in classic Roosevelt/Truman/Kennedy vision of a socialized America that doomed the TOS in the culture of cynicism and anti-Americanism that erupted in late 1960s.
In the original series, the socialistic themes are actually fairly muted, likely because in the mid-1960s, there wasn't much of a debate about big government in American just a debate about how big. Likewise, in STOS, you don't see much of the rest of the Federation. There are no politicians, corporations, or just plain folks. It's just Star Fleet, the odd colonist, artist on a couple of occasions, and, of course, Harry Mudd as the solitary representative of business people, who personified Roddenbury''s vision of the innate immortality of "greedy" business people compared to the altruistic agents of the State.
Ironically, it wasn't until the Reagan era, when culture shifted to become more optimistic about America, that Star Trek and it's optimistic themes became acceptable again.
On the other hand, still see Roddenbury's turn to the Socialistic vision gets more heavy handed, probably in semi-conscious counter reaction.
In "Encounter at Far Point" Roddenbury clearly adopted the post-60s cynicism about America in respect to the rest of the world. One of Q's manifestation is a pre-1970s American Marine Officer talking about patriotism and beating the commies, a stance which Picard mocks (by contrast imagine Picard mocking the same phrase with Nazis substituted for Commies.) In "Arsenal of Freedom", the entire premise of which centers around a leftwing view of defense contractors inevitably running wild. Neither have parallels in the more optimistic original series.
The real howler though is the introduction of the Frengi, who represent all the non-socialist in Roddenbury's vision. According to some, they were originally intended as primary villains of STNG.
In the beginning, the Frengi are every thing bad and nothing good. Revealingly, their entire culture concerns trade and commerce and in Roddenbury's vision that makes them not only evil but also viscerally disgusting. (BTW, the exact opposite of real-world trading-dominated cultures e.g. the Dutch Republic, which are open minded, tolerant, curious and prize honesty and self-sacrifice.)
And just to top it all off, physically, the Frengi are nothing but the centuries-long anti-Semitic stereotypes of Jews translated to space aliens. Anti-Semitism is grounded in the traditional hostility towards those who traded instead of farmed or conquered. Not only are they depicted as more or less genetically amoral, greedy, craven little liars who literally worship money, but physically the are short, swarthy-skinned with big ears and noses. All lifted straight from centuries of European depictions of greedy Jews. (Not that Roddenbury et al were consciously anti-Semitic, thats just the cultural template we all inherited from centuries of propaganda of what a greedy little business person is supposed to look like. When they set out to design a greedy alien, they naturally choose the existing cultural iconography, probably without thinking about it.)
Contrast this with the sympathetic treatment of the Klingons, an imperialistic race who worship war and conquest under the guise of "honor" and you see just how intensely a negative view Roddenbury and the other writers had towards those who engaged in commerce instead of politics and war. The Klingons are tragic, the Frengi vile. This is in keeping with the traditional Social social ranking, with intellectuals at the top, the masses in the middle, the military under them (if strictly needed) and the economic creatives down in gutter like pigs.
Star Trek, with its implied socialist society in the background, was optimistic as long as enough confidence in the Socialistic vision held in the audience, but as faith in Socialism and the State failed, so to fell the plausibility of the prefected Federation utopia. Star Trek themes concurrently grew darker and more morally ambiguous. Even in the STNG, you see a progressive darkening from the almost chirpy 1987 start, with the Soviet Union still carrying the Socialist banner, to 1992-94 after the Fall of Communism and the cultural reevaluation of socialism in all its strong and weak forms.
By the time DS9 really took off, Socialism was out of fashion and with it, all the easy answers it promised. Unlike Kirk or Picard, Sisko grapples with slippery practical and moral issues with no clear cut solution, right or wrong, or sometimes, even an objective reality. We start to see more the rest of Federation society and see that perhaps it isn't the socialist utopia long implied.
The biggest change came with the sympathetic treatment of the Frengi personified by Nog who, although he leaves private business to become a soldier of the State, is depicted using his trading skills for the common good. We are even given a peek at Frengi religion in which trade and commerce are seen as positive forces for the common good. This story line echos the reevaluation of positive role of Free Enterprise that occurred in the mid-90s throughout the entire world. After that point, the utopian vision of Socialism was dead as door nail and all the Star Treks since have played out under darker themes.
I really doubt we will see another optimistic fictional universe like the original Star Trek again until someone comes up with another utopian doctrine that can seize the greater imagination to a degree to let enough of the audience believe that a utopian society seems plausible at some point in the future. People have to be optimistic in the now, to both write and watch science fiction about an optimistic future.
Until then, it's probably more Captain Mel's than Captain Kirk's. Given the troubles the all the utopian Socialist caused, directly or indirectly, I'll gladly take the grubbier, smaller scale vision of Firefly in trade.