At the end of S9E12, the Doctor…
get all of his memories of Clara removed.
He does, however, keep a lot of surrounding information, some of which he lists near the end of the episode.
Does this include the time in the castle? That seems to be less related to what he forgot than some of the things he lists, but if he still knew that he spent such a long time there, he would surely want to know why. And if he does not remember, then that means that he only remembers less than 0.0001% of his lifetime (for some definition of "his lifetime").