Teams of children selected to end a war using nuclear weapons

Question

I am looking for the title and the author of (I think) a short novel. I read it as a teenager 30 years ago and I think it is much older. Here is a very approximate summary.

Teams of children are put though a series of increasingly difficult tests. They have to obey the rules, but the team we follow sometimes circumvents these rules.

In the final test, one can see on a screen (?) numerous enemy ships, their home planet and a few allied ships. It is a schematic representation. The only rule is: do not use nuclear weapons. But the team does not see any other solution and use nuclear weapons. This destroys the enemies and their planet.

One then understands (or actually one understands gradually through the text) that these tests are real. Adults just refuse to be held responsible for a genocide.

Answer 1

Could this be Ender's Game (or more specifically, the short novelette on which it was based, from 1977 edition of Analog Magazine)?

Ender is explicitly told that he can't use his weapon on the planet.

but does, because he's a cheeky little scamp.

Whereas in the full novel he's just told that he needs to make that decision himself.

Ender shrugged. Maezr began his explanation. “Today’s game, boy, has a new element. We’re staging this battle around a planet. This will complicate things in two ways. The planet isn’t large, on the scale we’re using, but the ansible can’t detect anything on the other side of it—so there’s a blind spot. Also, it’s against the rules to use weapons against the planet itself. All right?”

“Why, don’t the weapons work against planets?” Maezr answered coldly. “There are rules of war, Ender, that apply even in training games.”

Ender shook his head slowly. “Can the planet attack?”

Maezr looked nonplussed for a moment, then smiled. “I guess you’ll have to find that one out, boy. And one more thing. Today, Ender, your opponent isn’t the computer. I am your enemy today, and today I won’t be letting you off so easily. Today is a battle to the end. And I’ll use any means I can to defeat you.”

Answer 2

As has already been pointed out in the comments, this is likely Orson Scott Card's Ender's Game, but the issue is that this story comes in two versions:

• A novelette that first appeared in Analog in 1977, and has later appeared in many anthologies
• A 1985 novel-length work where Card added a bunch of framing for another novel he was working on, Speaker for the Dead

In the novelette, Ender Wiggins [sic] is the youngest commander at the Battle School, where children are trained to fight mock battles against a nameless alien species at war with humanity. At first there are zero-g laser-tag like games but then it changes to VR ship-based space battles (commanded by Ender). After the climactic battle in which life on the enemy's home planet is wiped out, it is revealed that at some point the space battles became real. Ender actually wiped out a whole species.

The novel-length work contains extra stuff:

• Ender's last name is "Wiggin", not "Wiggins"
• There is a lot of material about Ender's early life, especially his siblings
• Ender's childhood bullies are quite harshly dealt with
• The novelette's nameless, faceless enemy is described in the novel as an insectoid hivemind species called the "Buggers"

Since you mentioned a short novel, my guess is it's the original 1977 novelette, rather than the novel. But if you remember any of the later elements, it was the novel you read.