The Power Gem present in Guardians of the Galaxy appears to destroy everything organic that it touches.
This is demonstrated when one of The Collectors slave girls picks it up, the result of which is a large explosion after she appears to disintergrate. Additionally, Ronan himself can only hold the Power Gem for a very small duration of time before having to imprint it into his weapon - presumably, the Power Gem would have also overwhelmed him eventually thanks to the sheer amount of power it outputs.
The climax of the film sees Peter "Starlord" Quill surviving a significant period of time while holding the Power Gem. This is later hinted at being because of his biology - the Nova Corps mention that he is only half-Terran, and that they do not recognise the other half, claiming something along the lines of it being ancient.
Between different species of characters reacting differently to the Power Gem, and the Power Gem only destroying organic matter, I believe that resistance to the Gems power is granted by your biological make-up, rather than how "powerful" you are.
This is further supported by the fact that the Gem can be held for longer when multiple people join up - although unsuccessful when a group of people from one race tried it (as mentioned by The Collector), when Drax, Rocket Raccoon, Gamora and Starlord try it (all of whom are different species), they seem to be able to contain the power.
Additionally, it does not seem that the Power Gem destroys people in the comics.