Problems with Battleships that are arguably universal to space, sci-fi/fantasy space, and 20th century, 19th, 18th, 17th, 16th etc. naval warfare that contradict typical strategy/tactics of modern and perhaps futuristic military warfare and/or the goals of Starfleet in general:
It's a lot of eggs in one basket. You can cause a big problem in one place with a battleship. A battleship's worth of smaller craft can cause a whole lot of problems in a whole lot of different places or an almost equally big problem in one place. Smaller more mobile ships even with considerably less firepower if delivered against the same big target are going to tend to be more useful as long as their destructive/defensive capacity is good enough for a wider variety of concerns which they can most certainly achieve through greater variety. This applies to defense and offense.
The one thing they've generally been better at than anything else in their respective contemporary armed forces historically speaking is attacking stationary civilian targets on land from very far away. I'm not sure that translates to space battleships necessarily but as a classical weapons platform it's certainly not the sort of weapon of civilian slaughter and infrastructure destruction role Starfleet would want to emulate.
They're not necessarily slow but they've never been the fastest in the sea and it stands to reason they would never be the fastest in space in any drive system scenario where mass or volume is a non-trivial concern. If I loosely understand the rules of warp drive properly, mass is effectively reduced, not eliminated, and power drain is still relevant. You can't cause immediate problems for something you can't catch.
The more accurate and rapid-fire that weapons get, the less useful/versatile/cost-effective it is to have a whole lot of hugely expensive ones on one boat/starship. Damage to dollar, the US did serious damage with its PT boats while it was rebuilding replacements for its ships lost at Pearl Harbor. In a lot of ways the Defiant is similar in its focus to the PT boat although perhaps a much tougher bug to squash.
Battleships in a lot of ways were political weapons in much the same way nukes are today. That's another model one would hope Starfleet wouldn't be a huge fan of.
They are 100% offensive. You don't defend with a battleship. Smaller craft en masse can easily bypass and attack civilian targets through sheer force of numbers assuming relative costs are still roughly similar to what they would be now and in recent history in naval terms.
Now that said, this is sci-fi. Any one of the concerns I list could be imagined away through virtue of some circumstance or expense. If for instance one warp drive dwarfs all the other costs of building/maintaining a starship, it might make sense to have a lot more eggs in every basket. Star Trek however tends to try and find some level of analogy with modern naval forces I think and in that context we've decided there is no longer a practical use for such vessels either.
Full-Disclosure: I was a total swarmer in Homeword 2. This may inform some of my thoughts.