It isn't clear, but it is possible to make some informed guesses based on the very scanty information provided by the text.

>..."The oath that they broke was to fight against Sauron...  For at Erech there stands yet a black stone that was brought, it was said, from Númenor by Isildur... And upon it the King of the Mountains swore allegiance to him in the beginning of the realm of Gondor.  Nut when Sauron grew in might again, Isildur summoned the Men of the Mountains to fulfill their oath, and they would not; **for they had worshipped Sauron in the Dark Years**.   
- *The Return of the King, The Passing of the Grey Company*

These men had rejected Eru Ilúvatar, the one true god, and worshipped a false god;  this may have left them in a state similar to the Catholic concept of Purgatory or Limbo. This explanation would be consistent with Tolkien's personal beliefs as a devout Catholic.  The "rules" weren't actually broken by denying the Oathbreakers their right to go to Mandos;  the "rules" simply didn't apply to them yet, because of what we might call their "sins".  

On a more fundamental level, they were probably neither truly dead nor truly alive - they were somewhere in between the two.  As such, it is possible that they weren't eligible to go to Mandos yet.  

However, I think the most likely explanation is that their oath was seen as a willful rejection of their right to pass into Mandos at whatever time they ceased to be truly alive.  They could only reclaim this right by fulfilling their obligations as specified in their oath.