It's never been completely consistent, but the general idea is that the Holodeck tricks the people inside it, so they don't realize they're still in a small room.
To quote the relevant part of Memory Alpha's page on Holodecks:
Holodeck walls can generate holographic images that appear to extend
for an unlimited distance, seemingly much larger than its own
dimensions. In doing so, however, the holodeck is aware only of its
users; it does not recognize its own created objects. For example, if
a person were to throw a holographic rock at the holodeck's walls, the
rock would not be allowed to pass beyond the wall (if it were of
replicated matter). (TNG: "Encounter at Farpoint") It does this by
continuously adjusting the projections of the force fields and the use
of a force field "treadmill". With this, an individual approaching a
wall causes an instant shift away. It can also manipulate light
photons, 'lensing' them to make individuals appear further away if two
persons were separated in a scenario. The holodeck can change gravity
in three dimensions, so occupants don't notice the change, (Star Trek:
The Next Generation Interactive Technical Manual) as observed during
the stop of B'Elanna Torres' holographic orbital skydiving session.
(VOY: "Extreme Risk")
Additionally, it's shown that in some cases these tricks aren't foolproof, as in VOY 4x18-4x19, The Killing Game and The Killing Game, Part II, when the walls between Holodecks were broken down to make one huge hunting arena.