I can't remember if in the movies anyone explicitly states that Bifrost is the only way in and out from Asgard, but in Norse mitology Bifrost is just the bridge between Asgard and Midgard (Earth), it does not connect all Nine Worlds. Jötunheimr, for example, is just a different region on the same "planet" as Asgard, and they are separated by a simple river.