They swore an oath and broke it; an earlier plot-outline for the material (published in HoME 8) gives the basic text:
...the dark men of the Mountains, who swore allegiance to the sons of Elendil, vowing to aid them and their kin for ever, 'even though Death should take us.'
Oaths in Tolkien are powerful stuff. It was the oath of the Feanorians that led to the rebellion of the Noldor, the First Kinslaying, their return to Middle-earth, the wars against Morgoth, the Second and Third Kinslayings and Morgoth's final overthrow.
No doubt Isildur was aware of their oath (he may have been present when it was sworn) and his own curse probably amounts to not much more than "you said 'even though Death should take us' - so be it"; there doesn't seem to be much need to invoke any "magic" or "power" on the part of Isildur here; it needn't be much more than holding them to their oath.
So it's therefore more the swearing of the oath than the curse that led to their ultimate fate. Again, comparing with the oath of the Feanorians (from the Silmarillion):
They swore an oath which none shall break, and none should take, by the name even of Iluvatar, calling the Everlasting Dark upon them if they kept it not... For so sworn, good or evil, an oath may not be broken, and it shall pursue oathkeeper and oathbreaker to the world's end.
There's definitely a strong parallel between "calling the Everlasting Dark upon them if they kept it not" and "even though Death should take us", and note that such oaths "pursue oathbreaker to the world's end".
What's also interesting is the fact that only Iluvatar can change the fate of Men; this must therefore be an example of Iluvatar intervening in order to create a circumstance that would bring about the Dominion of Men.
It's interesting, but possibly not overly relevant to this question, that the Paths of the Dead itself was in origin an evil temple from the Second Age. The late essay "Rivers and Beacon Hills of Gondor" has this to say about it:
The Men of Darkness built temples, some of great size, usually surrounded by dark trees, often in caverns (natural or delved) in secret valleys of mountain-regions; such as the dreadful halls and passages under the Haunted Mountain beyond the Dark Door (Gate of the Dead) in Dunharrow. The special horror of the closed door before which the skeleton of Baldor was found was probably due to the fact that the door was the entrance to an evil temple hall to which Baldor had come, probably without opposition up to that point. But the door was shut in his face, and enemies that had followed him silently came up and broke his legs and left him to die in the darkness, unable to find any way out.
It may be interesting to ponder if the temple (no doubt used for Sauron-worship) had anything to do with the Dead Men's fate, but since Tolkien never seems to have written anything more about that particular topic, pondering is the best you can do.
