He outright broke Rule 1 and Rule 2.
to reveal themselves in forms of majesty, or to seek to rule the wills of Men or Elves by open display of power.
Unfinished Tales - Part IV: II The Istari
With Tolkien himself admitting Saruman's downfall.
Saruman the White, fell from his high errand, and becoming proud and impatient and enamoured of power sought to have his own will by force
Unfinished Tales - Part IV: II The Istari
The consequences of Saruman's actions were the worst possible outcome for the death of the mortal form of an Istari.
When his spirit looked to the west, to return from where it came. A wind blew it away in rejection. The rejected spirit is left to wander Middle-earth "house-less" being unable to take up a new physical form. For eternity.
To the dismay of those that stood by, about the body of Saruman a grey mist gathered, and rising slowly to a great height like smoke from a fire, as a pale shrouded figure it loomed over the Hill. For a moment it wavered, looking to the West; but out of the West came a cold wind, and it bent away, and with a sigh dissolved into nothing.
Return of the King: Book Six - Chapter VII, The Scouring of the Shire
It must be said. Gandalf had also broken the rules (revealing himself in a form of majesty against the Balrog). However, he did so to stop the ring from falling into the hands of a servant of Morgoth, in doing so he saved the ring. In that case, Gandalf was exempted. However he may or may not have been cast out of the Istari order for his actions and sent back to Middle-earth as a full-blown Maiar.