Darth Bane's vision for the Rule of Two was the gradual strengthening of the Sith Order over time, while playing to their strengths.
Bane's experience with the Sith on Korriban, and the Brotherhood of Darkness, showed him that one of the main weaknesses of the Sith was that the nature of the Dark Side of the Force made it inevitable that Sith would plot against each other in their pursuit of power.
By contrast, the Jedi would always work together. This put the Sith at a disadvantage whenever they attempted to directly confront the Jedi in organized conflict, as infighting and self-preservation would weaken the Sith while the Jedi remained coordinated and cooperative.
This doomed the Sith to always fail in open confrontation unless they could bring overwhelming force against their foe, yet Bane felt that overwhelming force was but one aspect of the Dark Side, and by rendering the more subtle Dark arts inconsequential, the Sith were not realizing their full potential. Subversion, manipulation, and corruption were aspects of the Sith powers that Bane felt were mostly being used against the interest of the Sith, rather than against the Jedi, especially given the current state of open military war between the two sides.
Furthermore, groups of weaker Sith would cooperate to undermine and destroy Sith stronger than them, which Bane saw as a fatal long-term flaw in the current Sith philosophy.
Darth Bane meditated long on these concerns, seeking a better understanding of the ways of the Dark Side.
Darth Bane's meditations, coupled with the revelations from Darth Revan's holocron, led Bane to develop the Rule of Two.
Having only two Sith, one to crave power, and the other to hold that power, would address all of Bane's concerns.
The Sith Lord taking on an apprentice would provide continuity to the Sith Order, allowing them to continue existing beyond one being's lifetime.
Having only one apprentice would ensure that the apprentice could only take the title of Sith Lord when he or she had learned enough to become stronger than their master, ensuring (theoretically) that each progressive Sith Lord would be stronger than the predecessor.
Being so few in numbers would also make it much easier to hide their presence from the Jedi, allowing the Sith to use the more subtle manipulations to achieve their goals, which Bane felt was one of the Sith's main advantages over the Jedi. Indeed, Darth Bane only enacted the Rule of Two once he had taken steps to convince the Jedi that the Sith Order was completely destroyed, to give him and his apprentice room and time to plan in secrecy.
Darth Bane was convinced that any Sith who didn't fully understand the superiority of his approach was fundamentally unworthy of the title of Sith Lord, and therefore was weak and needed to be destroyed. Indeed, he destroyed the entire Sith Order, and was only willing to accept an apprentice that understood this "truth".
The lessons of the importance of the Rule of Two were a central part of the training he provided to his apprentice, so that she, in turn, would choose her apprentice with those ideas in mind, and continue the chain.
So, to address some of the specific questions:
What is to keep any Sith from following the Rule of Two (or any rule, for that matter) if it is against their own self interest? If a Sith Master can stay in power longer by training a 2nd apprentice and pitting the two against each other, is he really going to worry about the Rule of Two more than his own plans and survival?
Two apprentices who are weaker than the Master can team up and destroy the Master, so having only one apprentice actually is safer for the Master, and therefore serves their self-interest.
Having one apprentice, as opposed to no apprentices at all, also serves the Master's self interest, because it allows them to exert direct influence on their plans while remaining removed from danger of discovery or direct conflict, in addition to giving them more flexibility in multi-tasking.
The Rule of Two also assumes that every Sith Master would place the survival of the Sith over his own survival, as opposed to attempting to ensure his own immortality in whatever way he could (like Plagueis). When you have a group that, by default, is arrogant and self-serving, as well as deceptive, why would they want the Sith to survive themselves?
Ascendency of the strong over the weak was one of the core tenets that Bane fervently believed in. As such, it is inevitable that any given Sith Master would eventually succumb to death, either from old age, or by someone stronger. At the time he created the Rule of Two, Darth Bane was not aware of any Sith technique to extend life, and certainly had no reason to believe that immortality (by any definition) was a possibility. Continuing the Sith beyond his lifetime was merely another aspect of how he exerted his power over the universe. If he could extend his life, he would, so I'm not sure that he really placed survival of the Sith over his own survival. Indeed, with only one Master, one could argue that survival of the Sith and survival of the Master were one and the same. Certainly no Sith Master was going to willingly let their apprentice kill them; that was counter to the very idea of the Rule of Two.
In fact, when Darth Bane started to feel the weakening of age, he was concerned that his apprentice would either never be strong enough to defeat him, or worse, that she was simply waiting until he grew old enough to weaken enough for her to easily defeat him. While under other circumstances, Bane would appreciate such a tactic, he was unwilling to let that happen, and started to look to ways to prolong his youth and health.
So does this rule, or any Sith rule, actually work? Or is it more of a suggestion that is broken when convenient?
As with any rule, the Sith follow it so long as it serves their interest. The fact that the rule survived for 1000 years is evidence that it actually worked.
