Federation starships come with plethora of powerful sensors and internal circuitry for long term exploratory assignments.... not to mention defensive and offensive systems, life support, etc.
That's probably the main reason why they are 'big' in the first place... or at least why most of their ships are as big (exploration and the need to house powerful technologies that would effectively make a starship self-sustaining for a given period)...
However, it is also accurate that as technology advances, you'll be able to create new and much more powerful technology with far less resources and physical space requirements.
I would imagine that in the 23rd century (at least during TOS) we could see internal corridors with some conduits, and would pack a larger crew compliment.
In the 24th century, technology advanced (obviously), took up less space, and you could create much more spacious quarters as a result... improving living standards for people on long term missions.
Mission parameters might be dictating the size of the crew for a given ship.
Or, a simple matter of crew assignments... so starships are built to a certain size, but crew compliments will vary depending on the mission they have... plus I doubt there's any particular need to fill starships to max capacity... especially if humanitarian aid is needed along with possible evacuation (which can also be handled if you keep some people in transporter stasis in addition to having some of them physically on the ship).
A Galaxy class can easily transport 10 000 people at any given time... but most of the time, it has a crew of just over 1000.
The Intrepid class for example might comfortably fit a crew of say 300, but only half as much were at first assigned for a simple mission to the Badlands as the mission parameters probably didn't require more (might also explain why Voyager launched with only 36 photon torpedoes).
So, its a combination of factors.
Ships are big due to being exploratory and suited for multiple things at the same time... Federation ships are also built to be relatively modular... so they can be easily upgraded over time.
Crew compliments would depend primarily on types of missions.
We saw the Defiant was a 'warship' and quite small.
If you automated the heck out of it, it would likely be able to run with only 1 or 2 people... with computer taking care of maintenance, repairs, upgrades, etc. (using transporters and replicators, or anti-grav drones which would have mini replicators, transporters and even tractor beams for moving stuff around).
Obviously, SF doesn't use extreme levels of automation in its ships (even though its more than doable to do this). I suspect, one of the reasons is to keep everyone well versed in ship functions... but quite honestly, this can be accomplished in a different capacity without having a rigid structure that forces people to do monotonous work all the time just for the sake of it (the needed skills can be maintained via gamification for example while they are doing something else... something more productive).