We don't know yet, whether the doctor is truly unaffected. Not a whole lot has happend since he met the parasite, but he was kneeling down and obviously not feeling so well having met that thing.
Yet, as the doctor makes a point of telling the parasite, he is so full of stories that we (and probably he himself) would never notice if he lost a few generations of experiences. If the Doctor was able to control which memories to give away, the effect would be even less.
There is also the possibility, that the parasite didn't really consume the Doctor's memories because it was afraid what it might see.
The Queens of Ages apparently only know songs, tales and other literature.
Real memories of what the Doctor has seen and done could be quite different (and harder) to digest than a thousand kids bedtime stories.
Last point. He is the Doctor and gets a kick out of visiting all those wonderful places in the universe. If he rids himself of some of his unspectacular travel experiences, he will get to visit all those places for the first time, again. I believe that would actually be quite appealing to him.