There are various types of kryptonite, and they all have varying effects on Superman. Each type of kryptonite contains/emits different kinds of radiation. I would think this is similar to the different radioactive elements contained within the planet Earth (plutonium, uranium, cesium, etc.)
We've been told in the comic books that green kryptonite, which is deadly to Superman, comes from the core of the planet Krypton. Likewise, the core of the planet Earth is radioactive and would kill any of us.
It stands to reason that if we were to end up on another planet, with a red sun, and if our cells were to consume the red sun radiation, giving us superpowers, that pieces of the Earth's core would still kill us - even if other beings on that planet were immune to it.
BTW - nothing in the comic books or movies or television shows says that kryptonite is exactly safe for humans either. But one would imagine that the rarity of the material generally keeps it from effecting us. If they were to find a vast kryptonite mine in Starling City or Gotham, my bet is that there would probably be a pretty large "cancer cluster" among the nearby humans.