A military threat is created when the population and the industrial potential of a species are diverted to military purposes. There is a proverb that in war, nobody is ever killed by a piece of gross domestic product, but it takes GDP to produce the means of warfare.
The utility of population vs. industry changed over history. Can a thousand men with bronze axes beat two thousand men with stone axes? Maybe. Maybe not. How about a hundred jet fighters vs. a thousand prop fighters? Yet there is no doubt that with an advanced technology like Star Trek, you don't need all that many people for your ships.
According to the Deep Space Nine Technical Manual, the Defiant had a maximum crew of 192; she could operate effectively with a much smaller crew.
- Imagine a society where 1% of the population serves in the space force.
- Imagine a planet with 5 billion inhabitants.
- They could crew over 260,000 Defiant-class ships.
Of course I haven't accounted for shipyards, and starbases, and headquarters staff, or for the people who build the tools to build the tools to build starships, but then there are 99% of the population left.
So it comes down to industry to build a quarter million Defiants, not to finding the crew. Industrial-scale replicators can help, but it is not feasible to replicate an entire starship.
How powerful is the industry of a single system? That depends. Do they rely on robotics? Most species in Star Trek don't. What about asteroid mining? Most species in Star Trek rather go interstellar and put their mines on a planet.
That being said, it seems clear to me that both Klingons and Romulans have more than one planet at their disposal. The difference is that in the Klingon and Romulan empires, only one species is in control, while the Federation is more democratic.