"Because Doctor Who" is pretty accurate
But we can look at it a little more closely and come up with a plausible explanation--just don't expect it to be what Moffat had in mind.
The Doctor is not supposed to cross his own time stream. No one is. It's one of the fundamental laws of Time Lord society, and there's a natural process --the Blinovitch Limitation Effect-- which kicks in (often disastrously) when the law is broken.
However, the Time Lords are able to mitigate the effects. In fact, that's how crossover stories like The Three Doctors and The Five Doctors were justified: Time Lord meddling with the Doctor's time stream, and compensating for the results.
The effects on continuity are troublesome. If you want to have the Second Doctor and Jamie show up in The Five Doctors (which you do, because they're awesome), you bring up awkward questions because there's no point in their known history where such a thing can fit! (Actually, there's a fan theory called "Season 6b" which is largely inspired by this particular discontinuity.)
There are two effects which might cause memory loss--or lack thereof.
The Blinovitch Limitation effect
In its natural state, time is not orderly and history is not fixed. Causality paradoxes are addressed by the Blinovitch Limitation Effect, which prevents paradoxes or --where they cannot be prevented-- tries to moderate their impact on history... often by making the offending person or object explode.
The Web of Time
The founders of Time Lord society found this unsatisfactory, and created the Web of Time, a "hitching post of chronology." It's a meta-structure imposed on history by the Time Lords, giving history a holistic and balanced form. The Web tries to automatically compensate when bits of history got fiddled with: Kill Hitler before his time and another man would take his place, save the Titanic and another ship will sink in its place.
Maintaining the Web of Time was one of the Time Lords' greatest self-assumed duties, and the Doctor's willingness to tweak the Web if it would save lives made him a historical terrorist in their eyes.
The Web is in disrepair now that the Time Lords aren't around to maintain it, but the presence of Fixed Points in History (as Ten was so fond of calling them) indicates that it's still working at some capacity. On the other hand, the massive continuity shifts that Eleven has managed show the Web isn't working very well.
These effects together can explain what's going on with the Doctor's memory.
Between the BLE and the Web of Time, there are competing forces acting to avoid and mitigate paradox. The BLE has a "burn it with fire" approach to causality problems, while the Web of Time has a gentler "smooth it over and get things back to normal" policy. The Doctor's conflicting memories and occasional forgetfulness is easily explained by the BLE and the Web struggling against each other (without the active influence of the Time Lords).