It's repeatedly referred to as "The Citadel at the End of Time" (or just "The Citadel") in interviews with the show's staff.
The Citadel at the End of Time exists on top of an asteroid. Speaking
to Marvel.com, Farahani reveals the inspiration, “In the comics,
there’s precedent for the Citadel at the End of Time being on an
asteroid,” referring to THOR (1966) #245. “What I proposed, early on,
was this idea that the entire building, all of the architecture, was
carved in situ from the asteroid; there were no other building
materials,” comparing it to carvings like Petra in Jordan. The Citadel
at the End of Time would be all carved from this black stone with gold
vein embellishments.
Farahani recommends fans to rewatch the series and spot the citadel
stone making its presence early on, pointing out that the statues in
Judge Renslayer’s office, the front of the judge’s dais in the Time
Court, and the elevator to the Time Keepers’ chamber, are all carved
from the same stone. “This is the link from He Who Remains to the TVA
— this rock that the whole place is quarried from,” says the
production designer.
The Citadel at the End of Time is meant to punctuate He Who Remains’
position as “this sad, lonely figure rattling around a big empty
space.” After all, the creator of the TVA had to stave off his other
variants, culminating in the Mutiversal War. Achieving cosmic harmony
is an arduous and solitary task. To reflect He Who Remains’ eons of
isolation, Farahani remarks, “The thinking with the Citadel was that
it was in ruins except for the office. He retreated from all the
different parts of the Citadel, abandoned them, and just holed up in
his office.”
Loki: Deconstructing He Who Remains’ Life’s Work at the Citadel at the End of Time
Interestingly, the structure seen in Thor Vol.1 #245 (mentioned above) was not the 'citadel' at the end of time, but rather a temple, hence the name of the story
"The Temple at the End of Time!".
