My guess would be simple: They would have to attack a 700 feet tall wall. Even if what Jon said, and they outnumbered the Brothers 100 men to 1, the Brothers would have a really really big advantage against the huge army: That huge, frozen Wall.
It is mentioned that quite a few Kings-Beyond-the-Wall have attacked Castle Black and the Wall before, but have failed. This must be why Mance is actually waiting and building a tremendous army before attacking the Wall.
(Not quite a spoiler, more like an explanation: ) In the books, it is said that Mance has united many of the clans that are North of the Wall. That didn't happen in one night. Mance didn't wake up one morning and had 100.000 followers. It must have taken a lot of time (years) to actually build an army that big.
If you have seen season 4:
You must have noticed that he plans an attack on Castle Black, first from the South and then from the North. That planning takes some time as well.
Also, think about this: An army of 100.000 people, doesn't move as easily and fast as 10 people on horses. A lot of Mance's followers are walking and not riding. This means that the march to the Wall would take way too much time.