Speaking exclusively from the MCU, it's hard to say (yet)
While we can't say definitively from what we've seen so far, we can potentially infer some truths from what the movies have shown us.
Of the stones encountered, all of them were able to be used/handled by when properly contained, while being less predictable when handled directly. The more powerful the being, the more likely they will be able to control/survive them directly.
Here's what we've seen:
- The Tesseract (space gem) could be utilized, but not directly held without consequences, at least by humans. I suspect Asgardians might be able to, although Thor only ever uses it in the container.
- The Chitari scepter (mind gem) was usable when within in the scepter, and outside by Vision (who might be regarded as sufficiently powerful enough).
- The Aether (reality gem) could be handled inside its container, but was at the very least harmful to others (and while relatively slow, was killing Jane when she was hosting it).
- The Orb (power gem) is probably the best example of the consequences of direct use by a being lacking sufficient power. Ronan would have been overwhelmed by direct contact after some time, but was able to bond it with his hammer to utilize it in a more controlled way.
- The Eye of Agamotto (time gem) we never see is direct contact with anyone. It's always utilized through the container that is the eye.
In all of the above examples, weaker beings (usually humans, but some others as well) are almost always overwhelmed by direct contact with an infinity gem. While sorcerers could potentially be an exception (depending on how "power" is defined, since so far it has always appeared genetic), from what we've seen so far there's a much better explanation.
Many of the housings for the gems are designed to utilize their power in a safe and controlled way (the Scepter, the Eye, and the power gem [in the hammer] namely, but the Tesseract to a lesser extent). Being that Strange has only ever used the time gem in this way, we can't really say he's truely "mastered" it at all, any more than Red Skull mastered the Tesseract, or Ronan the power gem.
So at present in the MCU, there's nothing as of yet that would allow us to know with certainty that sorcerers are capable of mastering even a single gem. Their human biology works against them, so in the end it's going to come down to how "power" is defined. I expect this will be addressed in Avengers: Infinity War, if not its sequel.