In Star Trek II: The Wrath of Khan, Khan explains to Chekov and Captain Terrell that
"Ceti Alpha VI exploded six months after we were left here. The shock shifted the orbit of this planet and everything was laid waste."
Why did it explode?
It's not explained. The closest we get is in the Vonda McIntyre novelisation. It was something to do with tectonic instability
"You lie!" Chekov shouted. "I saw the world we left you on! It was beautiful; it was like a garden—flowers, fruit trees, streams … and its moon!" Chekov remembered the moon most clearly, an enormous silver globe hanging over the land, ten times the size of the moon on Earth, for Captain Kirk had left Khan and his followers on one of a pair of worlds, a twin system in which planet and satellite were of a size. But one was living, the other lifeless.
"Alpha Ceti VI, our beautiful moon — you did not survey that, did you, Mr. Chekov? You never bothered to note its tectonic instability. It exploded, Mr. Chekov. It exploded! It laid waste to our planet. I enabled us to survive, I, with nothing to work with but the trivial contents of these cargo holds."
The To Reign in Hell: The Exile of Khan Noonien Singh novel by Greg Cox, about a later visit by Kirk & several of his friends to Seti Alpha V where they find journals describing Khan's & his group's time there, gives a couple of potential reasons. Starting on page 187 in the 2005 paperback version, Kirk mused about never understanding why the planet Seti Alpha VI exploded. Spock provides a reasonable possibility:
... "It is impossible to determine for certain, at least not without a comprehensive gravimetric analysis, but it is possible that a miniature black hole, perhaps ejected from a binary system elsewhere, passed through the Ceti Alpha system. Its tremendous gravitational pull could have literally torn Ceti Alpha VI apart while simultaneously affecting the orbit of Ceti Alpha V." ... "Such a disaster is theoretically possible, and fits the description provided by Khan."
When Kirk says they should've noticed this back then, Spock responds with:
"Not necessarily, captain. A black hole with mass sufficient to destroy Ceti Alpha VI might still have had an extremely small Schwarzchild radius." ... "If the hole was traveling through empty space, as it would have been en route to the Ceti Alpha system, there might have been very little evidence of its passage."
McCoy suggests another possible reason:
"What about some sort of artificial planet-killer?" ... "Like that doomsday machine we ran into way back when?"
However, Spock explains why it's unlikely:
... "Possible," he concluded, "but improbable. That particular mechanism was singularly methodical in its operation; it is unlikely that a similar device would have destroyed Ceti Alpha VI but spared the rest of the solar system."
Like Praxis, it exploded because the writers found it was convenient for it to do so.
Planets do not spontaneously explode. Planets / moons etc are essentially great rocks in space (thank you, Spock) and do not contain enough energy to overcome their own gravitational binding forces. Even if a satellite gets Death Starred the center of mass will remain in the same place. Praxis is absolutely laughable: a moon explodes, the shock wave nearly takes out a starship that is outside the solar system, and there's still someone behind to answer the radio? Right.
Far more likely is Ceti Alpha VI getting hit by something large, and the debris landing on V. Even moderate planetary bombardment will wreck your environment, and it will look like an explosion to anyone watching.
The odds against a planetary collusion in a system old enough to have multi celled life on one of the planets, are well - astronomical.
Planetary collisions are the type of crazy stuff which happens in young solar systems. But in the first few hundred million years bodies in unstable orbits are removed, spiraling inward and being adsorbed by the sun, or spiraling outward and being ejected from the solar system. And sometimes they collide with other planets and protoplanets when on their way in or out. After all the bodies in unstable orbits are removed that way in the first few hundred million years, the odd against any more planetary collisions are astronomical.
It took billions of years for multi-celled plants and animals to evolve on earth, so the Ceti eels prove that the Ceti Alpha system was billions of years old and thus the odds against any planets colliding and exploding would be astronomical.
But that would still be more more likely to happen than a planet exploding from internal forces. Exploding as a result of a collision would be more likely than exploding from internal forces by an amount that would also be - astronomical
Maybe Ceti Alpha VI was hit by the Nexus Ribbon (or something similar)? It travels through our galaxy every 39.1 years and would have passed through in 2254 and 2293. There was no mention of how long it takes to pass through the galaxy itself as it could have taken 13 years to get to Ceti Alpha VI after entering the galaxy.