If the entity (or entities as we see in The Guardians Of The Galaxy the whole team gets together and wields the power stone) wielding the infinity stone (depending on the stone) is powerful enough or worthy enough to wield it's power, then there wont be a problem.
In Captain America: The First Avenger the Red Skull touches the Tessaract with his bare hands and he gets teleported to another location by the Space Stone which is within the Tessaract, even though the Space stone wasn't directly held the stones are sentient and because the Red Skull tried to use it's power he was beamed away by the stone.
So not all stones will "fry people" but they do have their own safety mechanisms.
Also, in Doctor Strange we can see the time stone being wielded without any issues.
In Avegers: Age Of Ultron we see Vision having the Mind Gem on his head and even wielding it without any issues.
In avengers infinity war we see thanos holding many of the stones bare handed, as he is powerful enough he’s not affected by them.Also Doctor Strange in a way held the time stone just before he gives it to Thanos and we don’t see anything happen to Doctor Strange at that moment.
In the comics there were instance the stone might end up being shattered,
this happened when Captain America held the time stone in the comics,