For my part, I think this really was an "honest" psychological experiment by the Joker. It's not exactly the kind of thing that psychology post-docs can perform on the Psych 101 undergrads that have to participate for credit, so the underlying premise remains one of those open questions of human decision-making. The Joker obviously has no ethical compunction, so why not run the experiment in the realest way possible?
The way I always heard this experiment posited, the decision to make was not to commit murder, but suicide. The one I know prior to Dark Knight:
You and your soulmate are put in isolated, escape-proof cells where you can't see or communicate with each other. In each person's cell is a button. If the person in that cell presses their button, their cell will be flooded/gassed/immolated, killing them. If neither person presses the button in a specific time, they both die. If both press the button, they both die. If only one presses the button, the other person lives and is set free (but of course lives the rest of their life without their true love).
Do you press the button?
This wouldn't have worked as well in the movie because the people on the other boat are perfect strangers (and one boat was full of convicted violent criminals to boot). The choice for the civilian passengers is obvious, but it might be interesting to see if the hardened criminals show the better angels of their nature and opt for heroic sacrifice to save the civvies. But, the way the Joker set it up, they get that same chance, and it's now on the "sweet and innocent civilians" to show they're really fine upstanding moral people.
To answer your actual question, if one boat's passengers had pressed their detonator, I think the Joker would have been true enough to his word and left the other boat alone (though probably still adrift and helpless in the channel). He'd have gotten what he wanted out of the experiment; a big boom. It honestly wouldn't have mattered which boat did it. The ideal for the Joker would have been for the civilian passengers to do it, proving you don't have to be a career criminal to be a mass murderer. The second-best scenario would have been for the fine upstanding citizens of Gotham to be killed by the criminals on the boat because the civvies were too noble; pour encourager les autres.
As it turned out, neither side was sucked into the Joker's game. They instead were willing to let the Joker kill them all; it would be tragic, but it's on his head, not any of his victims. The look on the face of the civilian passenger who had the detonator in his hands says it all; had he actually done it, he could try to justify it all he wanted (they were convicts, he was saving people who'd done nothing wrong, he had no other choice but to kill or die), but the bare naked truth of it is that he would have played judge, jury and executioner of the lives of hundreds of other people (not all of which were convicted violent felons; remember the guards, and the ferry crew), just to save his own skin.