Only Ilúvatar has the power to give, or to deny, his gift. But oaths done in the name of Ilúvatar may have consequences beyond the power of the Valar. Fëanor's oath, as an example. Ilúvatar's will needn't to be perceived by men or elves clearly; his designs may appear in the history just as "Destiny" or "Doom". "Bilbo was destined to find the Ring".

There's no text evidence that the Dead Men of Dunharrow made his allegiance to Isildur in the name of Ilúvatar, but the consequences of his perjury say so. Isildur's power is not what doomed them, but "Destiny".