As noted elsewhere by Edlothiad, This is also mentioned in NEW AVENGERS (2013) #3, early in Jonathan Hickman's "incursion" storyline:
Apparently, the "limitations" mentioned by the Black Swan are what caused events later in the book
the Time Gem disappeared, and the other five gems seemed to be destroyed when Captain America used them to prevent the Earth from the other universe from colliding with ours, stopping this incursion.
Just in case you don't want to believe what a DC character says about a Marvel artifact....
NOTE: this may have relevance to the 2018 INFINITY event (presumably tying into the *Infinity War movie coming out), which has already shown that the Infinity Gems/Stones seem different than they did the last time we saw them. Since SECRET WARS in 2015, we are apparently in a different universe than we were (fairly firmly established in ULTIMATES and ULTIMATES2 after that event), so the stones being different kind of makes sense.