I can't say anything for the comics, but in the Marvel Cinematic Universe there is no way to know for sure.
However, we can take a look at the evidence presented to us.
There are two main possibilites;
- The "realms" are certain locations in the same universe, or
- The realms are entirely seperate dimensions
Realms are places evidence:
- When the Convergence happens in Thor: The Dark World, the portals to other dimensions seem to happen all over Earth and Earth alone - there is no mention of the Convergence in Guardians of the Galaxy.
- When Thor falls through one of the portals caused by the Convergence in Thor: The Dark World while summoning Mjolnir, it begins to shoot straight up into space, indicating that Thor has been simply moved to another location rather than dimension.
- In Thor: The Dark World, the Dark Elf ships are able to located and attack Asgard fairly trivially - if the Bifrost was required to access different dimensions, this shouldn't be possible.
- In Thor: The Dark World, Loki knows of at least one other way to leave Asgard without using the Bifrost or some kind of magic / highly advanced technology indiscernible from magic.
Realms are dimensions evidence:
- Asgard is unable to protect the other realms until the Bifrost is repaired after it's destruction in Thor.
- It is mentioned in The Avengers/Avengers Assemble that Odin had to use some kind of magic to get Thor to Earth - "How much dark energy did the Allfather have to use to conjure you here?".
After looking at the evidence, it seems to me that we can, at least for now, operate on the basis that the "realms" are merely certain physical locations within the same universe rather than different dimensions, as the evidence for the "Realms are dimensions" theory can be fairly trivially hand waved away - for example, Asgard being too remote a location to make non-Bifrost travel convinient.