Luck, timing (early in the outbreak), and the location of the farm (relatively far from major towns and cities). Walkers are attracted to people and noise, and the farm had little of either. There isn't much explicit canon information to explain this, but I read the comics and watch the show obsessively, I've read some of the novels, and I'm a zombie fanatic.
The farm wasn't spared completely though - many of Herschel's family members, friends, and neighbors were in the barn as walkers, because they were bitten. There are zombies in the area, but not as many as in other places.
Also, he did have fences - lots of them, one after another, to separate the fields and pastures from one another. A herd chasing a sound would (and did) break through them eventually, but a single zombie (or small group of them) wandering aimlessly would bump into the first line of fences and wander off in another direction.
Keep in mind that he group got to the farm maybe a month and a half after the outbreak started - maybe less - and left after a few weeks (Lori wasn't visibly pregnant when they left). There were lots of zombies around by then, of course, but not as many as there would be a year later. Think about season 1 - the group was camping within sight of central Atlanta, perhaps 5 miles or so outside of town, yet the first time we see a zombie enter the camp, someone says "They never come all the way up here!" or something like that. They were only a few miles from the highest concentration of zombies in the state, but they rarely saw any of them. They could even go into the city itself for supplies and make it back safely as long as no one fired a gun and attracted a herd. Herschel's farm is much farther from major population centers, so there were far fewer zombies in the area.
And as Cvance74's answer mentioned, the stream and woods acted as natural fences as well.
As for there being more zombies at the prison, several months had passed, so there were more zombies in the world. The prison was far from the farm and presumably closer to big towns and cities (they mention Macon in particular). And large numbers of zombies only start showing up after dozens of people from Woodbury move in, which made the prison a much noisier, busier place, with much more activity to attract zombies. Several pigs in a 2 acre yard behind 2 chain link fences attract more zombies than several cows on 200 acres behind a series of wooden fences. More animal noises in a more confined area that is more exposed and visible, basically.
Moving on to the issue of herds... I'll let Sgt. Abraham Ford explain (I edited out the naughty words).
Basically, walkers can't think at all, aside from the brain activity required to see/hear/detect prey (prey = living or recently killed people/animals), follow it, grab it, and eat it. Usually they ignore everything but potential food -if it isn't prey, or something that might be prey, they don't even notice it. If no such prey is around, they either hang out waiting for prey or wander aimlessly looking for prey. But sometimes, one zombie will see another zombie walking, assume that it is chasing prey, and starts following it. The walking zombie might indeed be chasing something, following a sound it heard, or it might just be wandering aimlessly - it doesn't matter. Then another zombie sees the first two walking, assumes that they are following prey, and starts following them. They pass another zombie, he assumes they are chasing something, ad he starts walking too... This goes on and on, and eventually you have a herd walking along chasing nothing, or maybe chasing a sound none of them remember hearing. They might keep walking forever - each one assuming that the others are following prey. Eventually, they'll probably stumble across some sort of prey, totally by accident.
Alternatively (this paragraph has its basis in both The Walking Dead AND the books World War Z and The Zombie Survival Guide), one zombie hears a scream (or a gunshot, or a car, or whatever). He moans, and starts walking towards it. Half a mile away, a second zombie hears the moan, moans himself, and starts walking towards the first moan. A quarter mile away, 2 more zombies hear the second zombie, the moan, and they follow the others... and so on and so on, until again, you end up with every zombie for 100 miles around walking in the same direction because of a sound only one of them heard.
Or a zombie might see a car or a person. He starts to follow the prey, but the car/person gets farther and farther ahead. The zombie will keep walking in the direction he last saw the prey going. Other zombies (that didn't see the prey) spot the zombie walking along, and they join him. As they pass other zombies, the group grows, and again, you get a herd chasing prey that none of them remember seeing.
This is what happened with the herd that attacked the farm. A few zombies saw a helicopter and started chasing it. It vanished out of sight very quickly, of course, but zombies are REALLY stupid, so they kept walking in the direction they last saw the helicopter going. Other zombies saw this group walking, assumed that they were heading towards food, and joined the group. They all forgot what they were chasing, but saw their herd moving and assumed that the other zombies knew there was food ahead, and this kept all of them moving in the same direction for days. Then, at exactly the right - or wrong - time, Carl fired his gun. The herd heard the shot and changed course - they turned and went towards the sound of the gunshot. If Carl hadn't fired the gun, they would have walked right past the farm. As I said, the herd attacking the farm was pure (bad) luck. And the fact that no herds had attacked it before that was pure (good) luck.