The question here seems to combine two separate things: 1. Is attending Quidditch matches compulsory? 2. Did anyone ever skip Quidditch simply of their own choice?
I think we can infer that Quidditch attendance is not mandatory from the fact that in Chapter Twenty-Four of Half-Blood Prince Harry has detention during a Quidditch match:
"Well, we shall see how you feel after your detentions," said Snape. "Ten o'clock Saturday morning, Potter. My office."
"But sir..." said Harry, looking up desperately. "Quidditch... the last match of the..."
"Ten o'clock," whispered Snape, with a smile that showed his yellow teeth. "Poor Gryffindor... fourth place this year, I fear..."
Given that we never find detentions occurring during mandatory student activities, it would seem to be reasonable to conclude that attendance at Quidditch matches was not, in fact, mandatory.
As for whether anyone ever ditched a Quidditch match of their own volition, Harry himself did this in Chapter Thirty of Order of the Phoenix:
“Listen,” he whispered, “can yeh come with me? Now?
While ev’ryone’s watchin’ the match?”
“Er... can’t it wait, Hagrid?” asked Harry. “Till the
match is over?
“No,” said Hagrid. “No, Harry, it’s gotta be now...
while ev’ryone’s lookin’ the other way... Please?”
Hagrid’s nose was gently dripping blood. His eyes
were both blackened. Harry had not seen him this
close up since his return to the school; he looked
utterly woebegone.
“’Course,” said Harry at once, “’course we’ll come...”
There are some implications over the following pages that they are doing something they shouldn't be doing. For example:
“I ’ppreciate this, you two, I really do,” said Hagrid as
they reached the stairs. He kept looking around
nervously as they descended toward the lawn below.
“I jus’ hope she doesn’ notice us goin’...”
And:
“Yeah,” said Hagrid. “C’mon now, quick, before we’re
spotted!”
And:
“Look — there’s people
comin’ out already — if you two hurry yeh’ll be able ter blend in with the crowd an’ no one’ll know you
weren’t there!”
But this may not be because missing Quidditch itself is problematic, but that the specific nature of what they were doing instead was problematic.