It has not been officially determined if the Marvel Cinematic Universe's Collector lives in OUR universe or is capable of traveling to Asgard under his own considerable power. However, since he will be appearing again in the Guardians of the Galaxy movie, it is safe to assume he lives in OUR (Midgard) universe.
In the Marvel Cinematic Universe

Little is known to the general public about the Collector, though he and the Asgardians appear to be aware of each other and generally friendly.
Benicio del Toro makes a cameo appearance as the Collector in a mid-credits scene of Thor: The Dark World. In the scene, Sif and Volstagg give to the Collector the Aether to keep in his collection, claiming they don't want to keep two Infinity Stones close together with the Tesseract already in Asgard. As they leave, the Collector remarks, "One down, five to go."
In the canon Marvel Universe (Earth-616)
The Collector is an alien extraterrestrial known for his psychological need to collect unique objects and species on his world-ship.

Taneleer Tivan is one of the Elders of the Universe and is close to his fellow Elder En Dwi Gast (the Grandmaster). He apparently came to self-awareness billions of years ago, on the planet Cygnus X-1. He is an enormously powerful being who wielded the Power Primordial and, though at first he took the appearance of an old human, his true form is a powerful alien.
For millions of years, the Collector lived on an unknown world with his wife and child, spending his days in thought and contemplation. Over three billion years ago, when his wife Matani lost the will to live and relinquished her immortality, the Collector realized he would need a hobby to maintain his own sanity, and began collecting interesting artifacts and life forms from around the universe.
Eventually, his obsession reached such heights he collected anything he considered rare or valuable often just for the sake of collecting. As such, he has a wide variety of rare or unique items at his disposal.
He is known as a member of loosely affiliated aliens as an Elder of the Universe. It makes sense to give the Aether to the Collector since he would have a very secure facility in which he keeps his collection of artifacts and sentients.
