The key is that Gandalf the White and Gandalf the Grey aren't quite the same person. The Gandalf that Aragorn, Legolas, and Gimli knew died, and the being that came back was similar, but not quite exactly the same; as Tolkien writes in Letter 156:
Gandalf really 'died', and was changed: for that seems to me the only real cheating, to represent anything that can be called 'death' as making no difference.
The Letters of J.R.R. Tolkien 156: To Robert Murray, SJ (draft). November 1954
He addresses this specific quote very obliquely later on in the letter:
The ‘wizards’, as such, had failed; or if you like: the crisis had become too grave and needed an enhancement of power. So Gandalf sacrificed himself, was accepted, and enhanced, and returned. 'Yes, that was the name. I was Gandalf.’ Of course he remains similar in personality and idiosyncrasy, but both his wisdom and power are much greater.
The Letters of J.R.R. Tolkien 156: To Robert Murray, SJ (draft). November 1954
Also, as multiple people have noted in comments on the question, there's really no way of knowing how much time as passed from Gandalf's perspective. The chronological time between his death and his resurrection is only a couple of weeks1, but what actually happened to him is unknown. At the very least he reverted to his Maiar form, which would have ripped open his consciousness and completely changed how he perceived the world; at most, he left the physical world and spent "time" outside of Time itself. Neither possibility bodes well for him remembering insignificant details like his name.
Gandalf's name
The question make an interesting (and incorrect) assertion that I want to redress (emphasis mine):
Why does he also give them permission to call him something he just recalled, a name he originally chose?
Gandalf did not choose this name, it was given to him. We don't know the exact circumstances, but Unfinished Tales tells us as much:
[H]e journeyed unwearingly on foot, leaning on a staff; and so he was called among Men of the North Gandalf, "the Elf of the Wand". For they deemed him (though in error, as has been said) to be of Elven-kind
Unfinished Tales Part IV Chapter II:"The Istari"
In any event, it seems unlikely that he would call himself "Elf of the Wand," since presumably he knows he's not actually an elf.
1 Per Appendix B, he died on January 25 and returned to life on February 14, and was picked up by Gwaihir on February 17
"Darkness took me. And I strayed out of thought and time."
. His several thousand years on Middle Earth would have felt like an insignificant amount of time compared to his trip, hence why he "forgot" his name.