"Double Divination this afternoon," Harry groaned, looking down. Divination was his least favorite subject, apart from Potions. Professor Trelawney kept predicting Harry's death, which he found extremely annoying.
"You should have given it up like me, shouldn't you?" said Hermione briskly, buttering herself some toast. "Then you'd be doing something sensible like Arithmancy."
Hermione's got a good point there.
WHY on earth would Harry continue taking that unscientific woo of a class? "Easy A"?
I reviewed Prisoner of Azkaban, Goblet of Fire, Order of the Phoenix, and Half-Blood Prince. Here's what seems to be going on.
I think Divination was compulsory through fifth year, until students were post-O.W.L.s¹. In Prisoner of Azkaban, Hermione is allowed to drop Divination halfway through the school year. Just as an exception was made to allow Hermione to use a Time Turner to attend ten classes, as I looked through the available info, it seems another exception was made and Hermione was allowed to drop a compulsory class. It would seem certain lessons were compulsory during years three, four, and five, but in order to move on to the N.E.W.T. level courses, minimum grades had to be achieved.
The distribution of timetables was more complicated than usual this year, for Professor McGonagall needed first to confirm that everybody had achieved the necessary O.W.L. grades to continue with their chosen N.E.W.T.s.
Half-Blood Prince - page 165 - Bloomsbury - chapter 9, The Half-blood Prince
If Harry could have dropped Divination during his third year, he surely would have -- he hated Divination with the passion of a thousand fiery suns! So this may be another indicator that Divination was a compulsory class.
I think the main point of J.K. Rowling purposefully showing us a lot of scenes of Professor Trelawney and Divination, and Harry's reaction to Trelawney and all that she represents was set up a guise that Trelawney was a fraud, inept, ridiculous, and ditzy. So when it's revealed that it was Sybill Trelawney who made the prophecy that the entire series is built around, it's a bit of a surprise and a Whoa! moment to both the reader and Harry.
A figure rose out of it, draped in shawls, her eyes magnified to enormous size behind her glasses, and she revolved slowly, her feet in the basin. But when Sybill Trelawney spoke, it was not in her usual ethereal, mystic voice, but in the harsh, hoarse tones Harry had heard her use once before:
‘The one with the power to vanquish the Dark Lord approaches ... born to those who have thrice defied him, born as the seventh month dies ... and the Dark Lord will mark him as his equal, but he will have power the Dark Lord knows not ... and either must die at the hand of the other for neither can live while the other survives ... the one with the power to vanquish the Dark Lord will be born as the seventh month dies ...’
Order of the Phoenix - page 741 - Bloomsbury - chapter 37, The Lost Prophecy
I think if all the pieces are put together, Harry took Divination in year four because it was 1) compulsory and 2) an important part of the build-up to the revelation of Trelawney's prophecy.
ETA: ¹At the end of their second year at Hogwarts, students are required to choose a minimum of two more subjects from the following list: Arithmancy, Muggle Studies, Divination, Study of Ancient Runes and Care of Magical Creatures. POTTERMORE - BOOK 1 - CHAPTER 6 - *Talons and Tea Leaves (screenshot).
So, when Harry picked Divination as one of his two elective classes, Divination became compulsory for him through the end of book five. As this is when the prophecy is revealed, and Harry earned a "P" (Poor) on his Divination O.W.L., there is no need for J.K. Rowling to continue showing the same scene of Harry hating Divination to the reader. McGonagall would never have let Harry into N.E.W.T.-level Divination anyway, even if -- for some reason that I can't even begin to fathom -- Harry had wanted to continue with Divination.
So, a small distinction, but one nonetheless: Divination was compulsory if it was a chosen elective. This is consistent with my existing answer, so I'll leave it as is.
Harry doesn't answer Hermione's question, and a clear answer is never given as to why Harry and Ron decide to continue with Divination in Goblet of Fire. For whatever reason, they do make a serious effort (spending hours trying to do it the right way, despite having little success. After returning "the old Divination class standby" of making stuff up, they did surprisingly well in the class:
Harry and Ron were deeply amused when Professor Trelawney told them that they had received top marks for their homework in their next Divination class. She read out large portions of their predictions, commending them for their unflinching acceptance of the horrors in store for them - but they were less amused when she asked them to do the same thing for the month after next; both of them were running out of ideas for catastrophes.
Harry and Ron were able to get by well enough, despite the fact that they hated the class. That's part of the reason they stick with it.
However, the bigger reason is that near the end of Chamber of Secrets, all the second year students must choose at least two of the following five subjects: Arithmancy, Muggle Studies, Divination, Study of Ancient Runes and Care of Magical Creatures. Harry and Ron both took only two (Divination and Care of Magical Creatures), so unlike Hermione, they didn't have the luxury of dropping Divination because they wouldn't have met the requirement of having two classes. It's never made apparent whether or not they could have transferred from Divination to another subject, but even if they could, they seem to have decided that Divination was easier than the other ones (although they might have chosen Muggle Studies, I suppose, unless Ron thought it would be hard or Harry thought it would be even more mind-numbing).
Out of universe, J.K. Rowling really liked the idea of having Divination as a subject at Hogwarts. She originally planned for Divination to be a first year subject that was mandatory, but dropped the idea later on (link here). Also, just about every prophecy, even some of the ones that Harry and Ron made up, came true, sometimes several books down the line.
I think the reason that Rowling kept making Harry take Divination is because she found it was a great way to make little foreshadowing hints of things to come. Harry and Ron hate it so much and Hermione thinks it's "wooly" in order to throw off the reader and make it so they don't realize there's something to it until later readings. And of course to give more time to Trelawny's character in order to set up the plot point that despite being seemingly inept, she created the prophecy about Harry and Voldemort's fate.
Harry does not drop out of Divination because he has no other options:
Take a different class instead
He can't do this because he would be a year behind everyone else, not having taken it in third year.
Take no class instead
He can't do this because fourth year students can't just have free periods. That's something that only starts in sixth year. If you could, then any lazy student would just not sign up for any of the additional classes.
Hermione didn't have this problem because even without Divination all her time slots were filled. She was only taking Divination in the first place by traveling back in time to attend multiple classes at the same time. As she says in the last chapter of Prisoner of Azkaban (my emphasis):
That Time-Turner, it was driving me mad. I've handed it in. Without Muggle Studies and Divination, I'll be able to have a normal schedule again."
