A logical reason would be peace. The Dominion lost acces to any form of reinforcements after closing the wormhole. Also, it's not uncommon for a cut-off army to make peace deals (what the Dominion did, even though it was only so they could regroup for a counter attack on Coridan, Benzar and Betazed), and when making a peace deal it is better to have some good standing with the enemy. The destruction of DS9 would be a big middle finger towards the Federation which is not something you want to do if you are about to enter negotiations.
A other reason would be the strategic importance, after DS9, the closest base of operations for the wormhole would be Bajor, and a planet is harder to control then a single starbase. The destruction of DS9 would force the Dominion to occupy a world that is known to resit occupation while for the Federation it would not be necessary to send an occupation force to use Bajor, due to their positive relationship, as a base meaning that in the long run DS9 would be more valuable for the Dominion then the Federation.
And at last, the Dominion clearly lost the battle with the Defiant standing ready to attack and a fleet of star ships on route, by announcing their retreat they effectively surrenders which caused the Federation to respect their retreat. If on the other hand the Dominion destroyed DS9 which still housed non-Dominion people it might have forced the Federation to be more aggressive on their retreating enemies.
This being said, if DS9 were Romulan/Klingon controlled they would probably have used the self destruct since neither of those have the close relations advantage with Bajor and would probably haven't shown the same mercy the Federation did.
