I think I understood that a rogue science experiment run by an Earth-based faction got out of control on Phoebe, perhaps on purpose.
Nothing got out of control on Phoebe; that happened on Eros. Phoebe was where the protomolecule originated.
Why? I thought Mars would want to capture it to prove that the UN was up to something bad there.
The answer to your main question: Mars likely could not capture the Phoebe research base, because they were outgunned by the UN ship the Nathan Hale, as explained to Draper by Sutton afterwards. Although it's used as the reason the Martians didn't fire at the Nathan Hale itself, UN could easily have stopped Mars from taking the station too.
Mars would probably have captured Phoebe if they had the firepower to do so, but they were outgunned and chose to do something the UN didn't expect instead: Destroy the base. After what happened at Eros, that was probably a good plan B.
The rest of this question contains some spoilers.
I guess the only explanation is that Mars knew what was going on there and did not want it known
I guess it was a joint research station so Mars possibly was involved in the experiment?
That's an explanation, and I can see how the UN could leap to the same conclusion, leading to war (as if there weren't already enough powder-kegs lying around that could also result in war).
Luckily, the UN learns in the next episode that Mars was not involved – but neither was the UN. It was a strictly private endeavour by Protogen.
Wouldn't the martians assume that UN had everything it needed from there already? What do they think they are denying the UN by blowing up Phoebe?
They don't know for sure, as they don't know about Thoth Station, but they assume they destroy the seat of protomolecule research and the research that enabled Eros to happen.