Are they allowed to retake the exam? Or would it mean they must repeat the same study year?
Or would it mean they must repeat the same study year?
Absolutely not. Fred and George for example got only three OWLs each, meaning they failed seven, but they certainly aren't held back:
'Anyway, it's a nightmare of a year, the fifth,' said George. 'If you care about exam results, anyway. Fred and I managed to keep our peckers up somehow.'
'Yeah ... you got, what was it, three OWLs each?' said Ron.
'Yep,' said Fred unconcernedly. 'But we feel our futures lie outside the world of academic achievement.'
'We seriously debated whether we were going to bother coming back for our seventh year,' said George brightly, 'now that we've got -'
Harry Potter and the Order of the Phoenix - p.205 - Bloomsbury - Chapter 12, Professor Umbridge
It only means that their choice of NEWT subject was pretty limited by their poor grades. This we can infer from the fact that Hogwarts teachers set entrance requirements for their classes. Snape, for example, famously required O at OWL for admittance into his NEWT Potions class.
Are they allowed to retake the exam?
Mithrandir's fine answer shows us that they are allowed but they certainly don't have to. There's no way Fred and George retook seven subjects in The Goblet of Fire (the year after their OWL year). Similarly, Harry failed Divination and History of Magic and did not retake them:
ORDINARY WIZARDING LEVEL RESULTS
Pass Grades: Outstanding (O), Exceeds Expectations (E), Acceptable (A)
Fail Grades: Poor (P), Dreadful (D), Troll (T)
HARRY JAMES POTTER HAS ACHIEVED:
History of Magic: D
Harry Potter and the Half-Blood Prince - p.100 - Bloomsbury - Chapter 5, An Excess of Phlegm
They either have to or can take them again the next year, presumably learning what they were supposed to the previous year, as we see Crabbe and Goyle do:
"Keep your voice down!" spat Snape, for Malfoy's voice had risen excitedly. "If your friends Crabbe and Goyle intend to pass their Defense Against the Dark Arts OWL this time around, they will need to work a little harder than they are doing at pres-"
Harry Potter and the Half-Blood Prince, chapter 15