In Star Trek II: The Wrath of Khan, Khan explains to Chekov and Captain Terrell that
"Ceti Alpha VI exploded six months after we were left here. The shock shifted the orbit of this planet and everything was laid waste."
Why did it explode?
It's not explained. The closest we get is in the Vonda McIntyre novelisation. It was something to do with tectonic instability
"You lie!" Chekov shouted. "I saw the world we left you on! It was beautiful; it was like a garden—flowers, fruit trees, streams … and its moon!" Chekov remembered the moon most clearly, an enormous silver globe hanging over the land, ten times the size of the moon on Earth, for Captain Kirk had left Khan and his followers on one of a pair of worlds, a twin system in which planet and satellite were of a size. But one was living, the other lifeless.
"Alpha Ceti VI, our beautiful moon — you did not survey that, did you, Mr. Chekov? You never bothered to note its tectonic instability. It exploded, Mr. Chekov. It exploded! It laid waste to our planet. I enabled us to survive, I, with nothing to work with but the trivial contents of these cargo holds."
The odds against a planetary collusion in a system old enough to have multi celled life on one of the planets, are well - astronomical.
Planetary collisions are the type of crazy stuff which happens in young solar systems. But in the first few hundred million years bodies in unstable orbits are removed, spiraling inward and being adsorbed by the sun, or spiraling outward and being ejected from the solar system. And sometimes they collide with other planets and protoplanets when on their way in or out. After all the bodies in unstable orbits are removed that way in the first few hundred million years, the odd against any more planetary collisions are astronomical.
It took billions of years for multi-celled plants and animals to evolve on earth, so the Ceti eels prove that the Ceti Alpha system was billions of years old and thus the odds against any planets colliding and exploding would be astronomical.
But that would still be more more likely to happen than a planet exploding from internal forces. Exploding as a result of a collision would be more likely than exploding from internal forces by an amount that would also be - astronomical
Like Praxis, it exploded because the writers found it was convenient for it to do so.
Planets do not spontaneously explode. Planets / moons etc are essentially great rocks in space (thank you, Spock) and do not contain enough energy to overcome their own gravitational binding forces. Even if a satellite gets Death Starred the center of mass will remain in the same place. Praxis is absolutely laughable: a moon explodes, the shock wave nearly takes out a starship that is outside the solar system, and there's still someone behind to answer the radio? Right.
Far more likely is Ceti Alpha VI getting hit by something large, and the debris landing on V. Even moderate planetary bombardment will wreck your environment, and it will look like an explosion to anyone watching.
Maybe Ceti Alpha VI was hit by the Nexus Ribbon (or something similar)? It travels through our galaxy every 39.1 years and would have passed through in 2254 and 2293. There was no mention of how long it takes to pass through the galaxy itself as it could have taken 13 years to get to Ceti Alpha VI after entering the galaxy.