There is a reference in one of the new series, (2005 onwards), that states that a rip in time and space called the Untempered Schism was used by the Time Lords in an initiation ceremony for children entering the Time Lord Academy.
At the age of eight, young Gallifreyans - not yet Time Lords - would be taken to the Schism. When subjected to its effects the children would react in different ways — "Some would be inspired, some would run away, and some would go mad." It is said that two of the Time Lords, one of these being the Doctor, agreed that the experience was "Painful".
It should be noted that not all Gallifreyans become Time Lords, but it is said that the "Untempered Schism" is what gave the Time Lords their ability to regenerate. Whether this applies to all Gallyfreyans or not is unkown.
I think though, that looking into the Time Vortex is enough to make a person lose some of their grip on reality. I surmise that it opens up their mind to the time stream and allows them to see what has been and what will be. Unfortunately this is a lot of information and its not always set in stone. I would think this can make remembering where you are and where you should be quite difficult. Things might get a bit mixed up.
The above might also account for the fact that the Doctor rarely arrives when he says he will. Some times he is early, and sometimes he is late. However, as a great man once said, (paraphrasing), "Time is relative and the perception of time is different for all of us."
That could explain why the Doctor sometimes perceives himself as being on time when others do not. In the great scheme of things time isn't linear, its a sort of "ball of wibbly wobbly timey wimey stuff".