At the end of Deathly Hallows Hagrid explains how he got back in to the school grounds:
Smashed our way through the boundary by the forest, Grawpy was carryin' us, Fang an' me.
I take this to mean that he did not simply come in through the forest with no physical obstacle. It would either mean that there is a boundary (wall) preventing one from entering the forest, or that they came through the boundary near the forest but there is actually no boundary around the forest. If the latter, though, why didn't they just come in through the forest without having to smash through the boundary? Hagrid of all people should have no trouble entering through the forest.
I think it is therefore likely that there is some sort of boundary wall preventing one from simply entering the grounds through the forest. This could either be a wall that encloses the entire forest, or it could be a wall that runs through the forest (i.e. if the forest is so big it is possible that part of the forest is on the school grounds and enclosed, but the rest of the forest extends unprotected).
Note also Aberforth's comment when the trio want to enter Hogwarts at the end of Deathly Hallows:
"There's only one way in now," said Aberforth. "You must know they've got all the old secret passageways covered at both ends, dementors all around the boundary walls, regular patrols inside the school from what my sources tell me. The place has never been so heavily guarded.
This too would seem to indicate that the forest is protected by the wall. Otherwise the dementors all around the boundary walls wouldn't really help if anyone could simply avoid them by going through the forest.
In any case, it seems clear that there is some sort of barrier around the forest, because throughout the series no one ever suggests the forest as a way to enter the grounds.
When, throughout Prisoner of Azkaban, everyone is trying to figure out how Sirius could get in, no one said "hey, maybe he came in through the forest". Similarly, in Half-Blood Prince Dumbledore asks Malfoy how he managed to smuggle Death Eaters in, and doesn't even consider the possibility that they just waltzed in through the forest.
However, at the start of term in Prisoner of Azkaban, Dumbledore makes the following announcement:
"They are stationed at every entrance to the grounds." Dumbledore continued,
Similarly, Hermione later says:
They're guarding every single entrance to the grounds.
"Every entrance" and "every single entrance" indicate that there are actually several entrances to the grounds, yet in the books we only see two entrances (the front gates, and the lake), not counting the secret passageways. Thus, there would seem to be other entrances that we are never told about. One of them might be the forest. So it may have a protective barrier, but also have an entrance. During the events of Prisoner of Azkaban and Deathly Hallows it would have been guarded by dementors, and perhaps by aurors during Half-Blood Prince.