Mance Rayder, the King-Beyond-the-Wall purportedly has an army of 100,000 men. Isn't this more than enough to crush the undermanned Night's Watch? Even Castle Black, their headquarters, has fewer than 100 men able to fight.
Even if he believed Jon Snow's inflated numbers, they still outnumber them 100 to 1. Or he could attack a weaker part of the wall, and go from there to Castle Black. What is Mance waiting for?
I haven't read all the books yet; please be mindful of spoilers
My guess would be simple: They would have to attack a 700 feet tall wall. Even if what Jon said, and they outnumbered the Brothers 100 men to 1, the Brothers would have a really really big advantage against the huge army: That huge, frozen Wall.
It is mentioned that quite a few Kings-Beyond-the-Wall have attacked Castle Black and the Wall before, but have failed. This must be why Mance is actually waiting and building a tremendous army before attacking the Wall.
(Not quite a spoiler, more like an explanation: ) In the books, it is said that Mance has united many of the clans that are North of the Wall. That didn't happen in one night. Mance didn't wake up one morning and had 100.000 followers. It must have taken a lot of time (years) to actually build an army that big.
If you have seen season 4:
You must have noticed that he plans an attack on Castle Black, first from the South and then from the North. That planning takes some time as well.
Also, think about this: An army of 100.000 people, doesn't move as easily and fast as 10 people on horses. A lot of Mance's followers are walking and not riding. This means that the march to the Wall would take way too much time.
Several reasons.
Remember that Mance isn't leading a conventional army. He is marching complete tribes and communities. So in addition to fighting men and women you have their families, children, beasts, and other worldly belongings. The logistics of making this huge mass of humanity (plus giants) move in the same direction is mind boggling, so you can't expect much speed.
And the Wildlings don't exactly have cutting edge siege engine technology. Castle Black is only one of nineteen castles spread more or less evenly across the Wall. While Mance knows for certain that the Watch isn't manning all of them, he has no way of knowing which ones are manned and which ones are not. Plus, the Watch mounts regular patrols atop the Wall. If Mance's slowly crawling army gets near they will be easily spotted and reinforcements called.
I don't know how far you've read so I won't elaborate more, except to say that Mance has people on the other side of the Wall ready and willing to make mischief.
In addition to the two correct answers and although not a direct answer to your question I'd like to add that as of Season 04 Episode 09, "The Watchers on the Wall", the wildlings have attacked Castle Black.
They did this in a very organised way by having a small group flank behind the Wall by climbing over it beforehand and then having the large army, which also contains Giants, attack front on. Even with such a large army and organised attack they were struggling to win, though it looked likely that they would. Until Stannis showed up to turn the tides to the favour of the Nights Watch again.
You can view the scene here:
