It's a yes and no answer, as the other user postulated. At various times, they've been portrayed as different dimensions, and as aspects of the same dimension. All are related to the tree Yggdrasil.

The tree is central to Norse mythology, and is depicted as having three roots. Each of the roots connects to a different level of realms.

1. First root (Asgard) - Consists of Vanaheim, Alfheim, Nidavellir. Valhalla is also depicted as being on this level.
2. Second root (Midgard) - Earthling level, Midgard and Jotenheim. From the maps linked, it shows that Midgard is reached via the Bifrost bridge, while Jotenheim is a separate ring (Much like Saturns rings), that is actually connected to the root.
3. Third root - Hel/Niffleheim - Lands of the dead. Hel is the land of the dead, and Niffleheim is the frozen land of the dishonored dead. The images are not clear as to if the root holds both, or just lands on Niffleheim.
4. Other realms - Svartalfheim/Muspelheim - These realms appear to only be connected by interdimensional passageways - Svartalfheim appears only connected to Midgard, although possibly to Asgard/Hel. Svartalfheim is home to the dark elves, while Muspelheim is home to demons.

Muspelheim and Jotunheim both appear connected to Asgard through passageways, as does Hel/Niffleheim and Hel/Asgard.

If you look at the poetic Eddas, they mention both six and nine worlds. The poetic edda mentions nine, connected by the "Glorious world tree that unites them." It lists six of them as

1. Menn - Humans
2. Vanir - Gods
3. Jotnar - Giants
4. Nair - Dead
5. Aesir - Gods
6. Alfar - Elves

The dark elves are stated as having their own world apart from the rest, and then the two primordial realms of ice and fire.

Map #2

It's a yes and no answer, as the other user postulated. At various times, they've been portrayed as different dimensions, and as aspects of the same dimension. All are related to the tree Yggdrasil.

The tree is central to Norse mythology, and is depicted as having three roots.

Marvel iteration

Each of the roots connects to a different level of realms.

1. First root (Asgard) - Consists of Vanaheim, Alfheim, Nidavellir. Valhalla is also depicted as being on this level.
2. Second root (Midgard) - Earthling level, Midgard and Jotenheim. From the maps linked, it shows that Midgard is reached via the Bifrost bridge, while Jotenheim is a separate ring (Much like Saturns rings), that is actually connected to the root.
3. Third root - Hel/Niffleheim - Lands of the dead. Hel is the land of the dead, and Niffleheim is the frozen land of the dishonored dead. The images are not clear as to if the root holds both, or just lands on Niffleheim.
4. Other realms - Svartalfheim/Muspelheim - These realms appear to only be connected by interdimensional passageways - Svartalfheim appears only connected to Midgard, although possibly to Asgard/Hel. Svartalfheim is home to the dark elves, while Muspelheim is home to demons.

Muspelheim and Jotunheim both appear connected to Asgard through passageways, as does Hel/Niffleheim and Hel/Asgard.

Norse Mythology

If you look at the poetic Eddas from the original, they mention both six and nine worlds. The poetic edda mentions nine, connected by the "Glorious world tree that unites them." It lists six of them as

1. Menn - Humans
2. Vanir - Gods
3. Jotnar - Giants
4. Nair - Dead
5. Aesir - Gods
6. Alfar - Elves

The dark elves are stated as having their own world apart from the rest, and then the two primordial realms of ice and fire. However, there has never really been a precise mapping. For example, Odin sent Hel to rule a region of Niffleheim, where the Marvel depicts them as two separate areas. The poetic edda also personifies the nine worlds as having nine giantesses, one for each realm.

Map #1 - Marvel representation

Map #2 - Modernized/enhanced fan interpretation of the Marvel map