The Bifrost is not the only way to reach Asgard, or the other realms, or else Asgard would have no reason to be afraid of anything.
As we see in the prologue of Thor, the Jotun are perfectly capable of reaching Midgard on their own. We can only assume they're capable of reaching the other realms as well. Similarly, in the prologue to Thor 2 we see Asgard struggling to restore order to the Nine Realms after the destruction of the Bifrost. Again, if the Bifrost was the only way to travel across realms, none of the others would have been able to cause trouble.
When, in Thor, the Jotun invade, Heimdall doesn't wonder how they got there. He wonders how they got there without him seeing them. It wasn't a matter of having Jotun arrive on Asgard; it was a matter of having them inside Asgard with no warning of their approach.
The implication is that the Bifrost is merely the most direct way to travel between realms. The other methods are more "old fashioned". That is, having the Bifrost is a strategic advantage to Asgard, allowing them to move troops into and out of the other realms directly and immediately, with exact placement and quick retreat. Imagine, for example, how pre-air-flight armies would view the ability to drop paratroopers: you could get your soldiers to their destination by boat or tank, but being able to fly them overhead and put them exactly where you want them is a huge benefit.
If only one realm - Asgard - has this strategic advantage, it would put them in exactly the position of power we see in the movies.