It could be anywhere. The Collector has hundreds of worlds and museum-ships which store his various collections of plants, animals, and exotic artifacts he has collected over his career.
Since he appeared to be more annoyed than distressed, it can be assumed the Infinity Stone is still safe.
The Marvel Cinematic Universe has not shown the Collector to be the cosmic being he is in the canon Marvel Universe. To understand the Collector, you have to understand WHY he collects things.
The Collector, Taneleer Tivan is one of the Elders of the Universe. He apparently came to self-awareness billions of years ago, on the planet Cygnus X-1.

He is an enormously powerful being who wielded the Power Primordial and, though at first he took the appearance of an old human, his true form is a powerful alien. The Elders of the Universe are some of the oldest beings in the Marvel Universe. To maintain their sanity over the millennia, each has dedicated themselves to a particular obsession.
The Collector collects. Everything. Plants, animals, exotic technology, even sentient life forms. He even tried to collect the Avengers once or twice.

Avengers #119
He even uses time travel to find creatures and objects from the pasts of the civilizations he has visited.
This is the most important thing to remember: He has been collecting things for MILLIONS of years.
He has filled up entire worlds with his collections. He also has museum-ships where he stores his most precious objects. The idea that he would keep the Aether on Knowhere (a planetoid filled with reprobates) is highly unlikely. Knowhere looks more like a trading post than a fully fledged collection.