Luck, timing (early in the outbreak) and the location of the farm (relatively far from major towns and cities). Walkers are attracted to people and noise, and the farm had little of either. There isn't much explicit canon information to explain this, but I read the comics and watch the show obsessively, I've read some of the novels, and I'm a zombie fanatic.
The farm wasn't spared completely though - many of Herschel's family members, friends, and neighbors were in the barn as walkers, because they were bitten. There are zombies in the area, but not as many as in other places.
Also, he did have fences - lots of them, one after another, to separate the fields and pastures from one another. A herd chasing a sound would (and did) break through them eventually, but a single zombie (or small group of them) wandering aimlessly would bump into the first line of fences and wander off in another direction.
Keep in mind that he group got to the farm maybe a month and a half after the outbreak started - maybe less - and left after a few weeks (Lori wasn't visibly pregnant when they left). There were lots of zombies around by then, of course, but not as many as there would be a year later. Think about season 1 - the group was camping within sight of central Atlanta, perhaps 5 miles or so outside of town, yet the first time we see a zombie enter the camp, someone says "They never come all the way up here!" or something like that. They were only a few miles from the highest concentration of zombies in the state, but they rarely saw any of them. They could even go into the city itself for supplies and make it back safely as long as no one fired a gun and attracted a herd. Herschel's farm is much farther from major population centers, so there were far fewer zombies in the area.
And as Cvance74's answer mentioned, the stream and woods acted as natural fences as well.
As for there being more zombies at the prison, several months had passed, so there were more zombies in the world. The prison was far from the farm and presumably closer to big towns and cities (they mention Macon in particular). And large numbers of zombies only start showing up after dozens of people from Woodbury move in, which made the prison a much noisier, busier place, with much more activity to attract zombies. Several pigs in a 2 acre yard behind 2 chain link fences attract more zombies than several cows on 200 acres behind a series of wooden fences. More animal noises in a more confined area that is more exposed and visible, basically.