The most specific reason given by the Jedi in opposition to training Anakin (beyond merely stating that he is too old) is:
He is too old. There is already too much anger in him.
Mace Windu, Star Wars: Epsiode I The Phantom Menace
Anakin has had time to grow up with the injustice of slavery for both him and his mother, and this is just one reason why he's developed some anger which makes him more likely to turn to the dark side.
Although it's possible that Anakin might have turned to the dark side even without training, at least in that case he would be less dangerous because he would have much less command of the Force. Of course, the Jedi would have to keep an eye on him to make sure he didn't meet a Sith or some other Dark Jedi and learn about the dark side from such a person. The same is true of any other Force-sensitive individual who was too old when discovered -- it's better to leave them untrained than risk training them to have good command of the Force and then turn to the dark side.
Another reason why age is a factor is that it takes a long time to complete Jedi training. Obi-Wan, for example, started training "from a very young age" and didn't even become a Padawan until he was about 13 years old. Anakin essentially skipped all this training and became a Padawan directly, but this probably only possible because of his record high midi-chlorian count. It would take too long to train most Force-sensitive individuals if they started at Anakin's age.
(For the more cynical among us, the Jedi also need to "get 'em while they're young" in order to more effectively
brainwash indoctrinate the trainees in the Jedi teachings.)