I've come up with several plausible reasons.
With regard to NOMAD (and also, presumably where similar circumstances occur in TNG : The High Ground and DS9: Trials and Tribble-ations), the simple fact is that the transporter is specifically geared not to allow objects to be destroyed, except under exceptional circumstances and not without considerable effort on the part of the operator. When trying to beam a bomb into space, you would presumably want to have as few steps (and hence as little room for error) as humanly possible.
The TNG Technical Manual notes that there are no less than two different safety features that must be overriden in order to disengage the confinement beam although it does mention that this technique can be used on dangerous substances like bombs if you wish to simply destroy them. My guess is that this is something you would do if you had slightly more time to prepare.
Disengaging the annular confinement beam will cause the materializing matter stream to have no reference matrix against which to form. In
such a case, the transport subject will form in a random fashion,
usually taking the form of randomly dissociated gases and microscopic
particulates. Operator override of the transport autosequence can
cause the ACB to be deactivated in order to allow the harmless
dispersal of a highly dangerous transport subject such as an explosive
device. Two safety interlocks prevent this option from being
accidentally activated. Such dispersal is usually accomplished by
transporting the subject into space.
NOMAD is paranoid
NOMAD has shown that it is extremely reactive to threats against its person. It vaporised the two guards that attempted to block its movement and killed Scott for merely touching its energy screens.
Overtly telling Spock or Scott to beam it out in a dispersed pattern might be taken as an act of aggression and result in NOMAD attempting to defend itself.
They don't want to destroy NOMAD (at least not to begin with...)
When they first encounter NOMAD, they simply don't believe that a ship that small could possibly be responsible for the destruction they've encountered. They're also under the impression that the probe actually houses little tiny people. NOMAD has also signalled that its intent is non-hostile. Starfleet isn't in the business of murdering people:
NOMAD [OC]: USS Enterprise, this is Nomad. My mission is non-hostile.
SCOTT: What kind of intelligent creatures can exist in a thing that small?
SPOCK: Intelligence does not necessarily require bulk, Mister Scott.