Were there any dementors in Nurmengard at all? If there weren't, it seems ironic that one of the most darkest wizards, Grindelwald, was being kept in a relatively less-punishing place, unlike Azkaban where all prisoners, irrespective of the severity of the crime, are exposed to the horror of the dementors.
There were no Dementors in Nurmengard.
It should be noted that the Dementors only resided in Azkaban because it was a natural breeding ground, even before the site became used as a prison.
Those who entered to investigate refused afterwards to talk of what they had found inside, but the least frightening part of it was that the place was infested with Dementors...The very walls of the building seemed steeped in misery and pain, and the Dementors were determined to cling to it. Experts who had studied buildings built with and around Dark magic contended that Azkaban might wreak its own revenge upon anybody attempting to destroy it. The fortress was therefore left abandoned for many years, a home to continually breeding Dementors.
So we wouldn't expect Dementors in Nurmengard anyway.
The details we have on Nurmengard are few and far between. Whether its design appears effective to us or not, it obviously was effective as it contained one of the most feared dark wizards for over 50 years. If your most dangerous prisoner dies inside the walls then your prison design must be sound!
How did it contain its prisoners? We only get snippets of description from the books but what we do get is suggestive.
...he was gliding around the high walls of the black fortress -
...looking up, up to the topmost window, the highest tower -
...time to fly -
...and he rose into the night, flying straight up to the window at the very top of the tower -
...the window was the merest slit in the black rock, not big enough for a man to enter...a skeletal figure was just visible through it, curled beneath a blanket...dead or sleeping...?
...as he forced himself through the slit of a window like a snake and landed, lightly as vapour, inside the cell-like room -
(Deathly Hallows, Chapter 23, Malfoy Manor).
From this we can deduce that Nurmengard is a tall, black tower which keeps its most dangerous prisoners at the top. It does have windows but these are mere slits.
The most powerful defence that Nurmengard has going is shared by Azkaban: it denies its prisoners their wands. Whilst some rudimentary magic is possible without a wand this really is extremely limited. Voldemort got through the window, but only with a wand. Grindelwald never managed to, probably because he couldn't manage that feat without a wand. Even if he were to somehow blast a hole in the wall or slide through using magic he faced a huge drop. This could be survived by using a Cushioning Charm (as demonstrated by Hermione in Gringotts), but not without the use of a wand. Like Azkaban, Nurmengard relies on its geography. Without a wand, Grindelwald would have been in for a nasty fall.
Presumably there were measures in place which prevented outside interference to break prisoners out. There were probably guards around, although they're never mentioned and Voldemort doesn't come across them. Otherwise, Grindelwald's followers would probably have flown in on broomsticks and set him free.