TV Tropes calls this Oh Look, More Rooms!
A subtrope of Bigger on the Inside. A place looks fairly normal from the outside, and possibly even when you get in, it's of a reasonable size. But there's this door in the back. Open it, and... there's a whole new section of the place, easily as big or bigger than everything you've seen so far! And look - there's a door in the back of that too, which leads to yet another new section - or worse, five doors...
Basically, it's when Bigger on the Inside keeps happening to the point of an Overly Long Gag. Related to Big Labyrinthine Building. Can be especially nightmarish in the Mobile Maze, where you can also know that the rooms weren't there before.
The examples listed on this page under Literature include the House of the Undying from A Song of Ice and Fire, and also the Temple of a Thousand Rooms from The Neverending Story, which was the example that sprung to my mind immediately from your explanation.