Mutually agreed treaties between sovereign powers (for example this one) don't usually have penalties attached unless they've been imposed on one party by another, typically following a military defeat.
Since both the United Federation of Planet and the Romulan Star Empire agreed the 'Treaty of Algeron' in good faith if either party intentionally broke the treaty, it might constitute a justifiable Act of War but that's it.
For the record, the full terms of the Treaty of Algeron have never been codified in any of the trek shows or films (or even the EU books) so all of this is pure guesswork.
As regards 'individual' punishments for breaches of the Treaty; Commander Riker and Admiral Pressman were both placed under arrest at the end of the "Pegasus Incident". Apparently Riker and Picard suffered no further punishment for their actions, presumably because Riker claimed to be young and stupid and Picard claimed exigent circumstances.
The TNG Short Story : Orphans reveals that Admiral Pressman was ultimately court-martialed for his actions and dishonourably discharged. He subsequently became a senior member of Section-31.
On the other side of the fence, Commander Tomalak repeatedly violated the Neutral Zone and evidently suffered no punishment by the Romulan Empire.