Talos IV is a very dangerous planet that has lots of appeal to visitors.
Prior to The Menagerie, here’s what Starfleet know about Talos IV:
- The planet is very attractive. The Talosians were a very advanced civilisation who have been driven underground. That means lots of lost Talosian tech is sitting around for somebody to go and find.
- The planet is very dangerous. They can control anybody who pays them a visit. That means that if they were intent on universal domination, it would be easy – as long as somebody gives them a spacecraft. They could control the occupants, and use them to infiltrate and conquer the Federation.
This gives Starfleet a headache. They need to cut off the planet, and kill anybody who has (or might have) visited. How do you do that?
An absolute death penalty is a pretty good solution. It’s a strong deterrent to visiting, and makes it easy to justify killing anybody who survives the trip.
You have to be this extreme, because softer solutions don’t work. You can’t set up a blockade, because the Talosians could capture the blockading ships. You can’t explain why Talos IV is dangerous, because plenty of people would want to capture the illusion technology for their own ends. And a ban without teeth won’t deter many people – if anything, it might make the planet more attractive.
(And Starfleet would never trust promises of benevolence, or a supposed lack of ambition. The risk of complacency is too high – if you’re complacent, and you’re wrong, then the Federation is lost.)
Most other species don’t pose the same degree of threat, or aren’t as difficult to contain. The Talosians seem to be an unusually potent and dangerous enemy.
It’s worth noting that the death penalty was later lifted. In Turnabout Intruder, we’re told that the only death penalty is General Order 4. That’s because of the events of The Menagerie, where we see that the Talosian threat is minimal.
Kirk learns that the Talosians can’t be contained by avoiding the planet. The illusory Mendez – who appears light-years away from Talos IV – serves as proof of this. Starfleet’s existing containment tactics have failed. And yet, they haven’t taken over the universe, or at least don’t appear to have done so. (Either they’re really harmless, or they’ve already won and taken over everyone and everything – in any case, there’s nothing you could do.)
The death penalty is an exceptionally unusual order, reserved for the gravest of threats. And when it no longer necessary or relevant, it’s lifted appropriately.
(It’s been years since I watched any ST: TOS, so this is based on sketchy memories and Memory Alpha. Please point out any mistakes in the comments.)