I recently read about the structure of the nine hells or 'Baator' in the Forgotten Realms setting. I was wondering how large each of the different layers actually are. I mean, are they endless or are they similar to continents or planets in this plane of existence?
Go with nigh-infinite and for all intents and purposes endless.
Forgotten Realms cosmology of hell
This is one of the most complicated topics in most Roleplaying games and in the prose stories of the multiple realities of the Forgotten Realms. Cosmology in the Forgotten Realms postulates that each plane of existence is a nigh-infinite place centered around a central palace or place of power where the supposed ruler of that plane lived.
The Nine Hells were the home of lawful evil creatures commonly called devils who battled for control of their respective hells. Unlike demons, who were very chaotic in their battles, devils used more subtle means, assassinations, coups, and other schemes to gain power. Each of the nine Hells had its own physical laws or properties of matter but all were inhospitable or deadly to outsiders.
Stories (and RPG modules) tend to overcome the infinite nature of such places by having characters venture to very specific landmarks or events which focus the story AWAY from the nigh-infinite nature of these planes of existence.
Some planes might have more than one "ruler" in which case the center of the plane was in dispute.
The Great Wheel cosmology model placed the Nine Hells in the Outer Planes between Gehenna and Acheron, with additional connections to Concordant Opposition and the Astral Plane.
Each Hell was a different infinite layer interconnected at barriers much like a nine-layered cake—the lowest points of one layer manifested barriers that exited high above the surface of the next lower layer. The river Styx flowed through the first layer, Avernus, and also the fifth layer, Stygia, before crossing over into Gehenna.
An older image of the inverted layer cake of Hell can be seen here. Despite the cake-like appearance, each of these planes is a complete and separate universe/continuum with its own magical "laws" and "physics/physical properties".
This topic cannot hope to be covered in any depth here, so please frequent the links above for a better understanding of these overall explanations. All of the links are to the Forgotten Realms Wiki which does an adequate job of explaining them in detail.