When the Oathbreakers did not come to Isildur's aid as they had sworn to do, Isildur cursed them to be bound to Middle-Earth, unable to leave even after death until they fulfil their oaths to his line.
The Gift of Man is Illuvatar's blessing to grant to all mankind, good or evil. How is Isildur able to deny Illuvatar's gift to the Oathbreakers simply because of some earthly oath made to a royal Númenorean who possesses no divine power?