In Serenity, Mal first fights the Operative at Inara's home. Mal and Inara both get their butts kicked; the Operative finds a sword and starts making a big speech before he goes for the kill, when Inara surprises him by saying:
and that's not incense.
Boom! Flash bomb.
Mal and Inara escape, and the Operative is found lying on the ground surrounded by Alliance soldiers.
Mal had his gun. There was also a sword lying around somewhere near the Operative. Mal also had his hands as a weapon (to choke the Operative to death).
If Mal knew that the Operative was hunting down River and was obviously prepared to kill anyone who got in his way, and also had time to escape, why didn't Mal just kill the Operative right then and there?