Picard ordered the ship to be destroyed in the way that a POW may commit suicide to avoid being tortured by the guards; Nagilum was going to kill one-third of the crew in various painful ways, which would definitely qualify as torture. By killing his whole crew in a relatively quick explosion, Picard avoided having one-third of his crew suffer painful deaths. This may have been worth the extra lives to him, in the same way that a doctor might euthanise a patient. It's also a fairly standard Star Trek response to treat any form of captivity with contempt; it goes as far back as The Cage, where Captain Pike preferred to be tortured psychically by the Talosians than to ever give in to their desires, including threatening a suicide pact with the three women on the planet with him if they weren't released.
This may also have been done as a way to show Nagilum, an immortal being, exactly how much humans valued life, which was the reason for its experiments. While this is not stated in the episode, Picard's decision may also have been a nod to Kirk's actions in The Corbomite Manoeuvre; a bluff to ensure the survival of his whole crew.