Because the Dursleys hated magic more than they hated Harry. And as time passes they do like him going to school. Remember that they do want him gone in the seventh book.
Petunia's reason: Jealousy; and she wanted no reminder of how she was treated "unfairly" as she was not allowed to go to Hogwarts which broke her relationship with her sister and made her bitter. After her sister died she probably wanted to forget the whole magic thing as it only caused her pain:
She spent her childhood never getting over that slight at not going to school.
And her sister was dead.
She also might have been scared of magic.
Pg. 53 of Sorcerer's Stone, The Keeper of the Keys
'Knew!" shrieked Aunt Petunia suddenly. "Knew! Of course we knew! How
could you not be, my dratted sister being what she was? Oh, she got a
letter just like that and disappeared off to that - that school- and
came home every vacation with her pockets full of frog spawn, turning
teacups into rats. I was the only one who saw her for what she was - a
freak! But for my mother and father, oh no, it was Lily this and Lily
that, they were proud of having a witch in the family!"
She stopped to draw a deep breath and then went ranting on. It seemed
she had been wanting to say this for years.
"Then she met that Potter at school and they left and got married and
had you, and of course I knew you'd be just the same. just as strange,
just as - as - abnormal - and then, if you please, she went and got
herself blown up and we got landed with you!"
Vernon's reason: I think that in keeping Harry from going to school Vernon was the real mastermind. He was the one who stopped or tried to stop Harry from going to school most actively. I think that Petunia would have caved and let Harry go to school if she hadn't been trying to please her husband.
Vernon wanted a non-freak family
Chapter 4, The Keeper of the Keys, pg.53
"We swore when we took him in we'd put a stop to that rubbish," said
Uncle Vernon, "swore we'd stamp it out of him! Wizard indeed!"
Chapter 17. Sorcerer's stone, The Man with the two faces.
It was Uncle Vernon, still purple faced, still mustached, still
looking furious at the nerve of Harry, carrying an owl in a cage in a
station full of ordinary people.
Chapter 1, Prisoner of Azkaban, Owl Post
"HOW DARE YOU GIVE THIS NUMBER TO PEOPLE LIKE - PEOPLE LIKE YOU!" Uncle
Vernon had roared, spraying Harry in spit.
He was scared of the unknown. He hated not understanding.
Chapter 3, The Dursleys departing
"Well, I don't believe it," repeated Uncle Vernon, coming to a halt in
front of Harry again. "I was awake half the night thinking it all
over, and I believe it's a plot to get the house."
So he creates a outrageous tale.
And the whole of chapter 3 is basically Vernon trying to understand the situation and him wondering how he can manipulate the situation.
And he was interested in not paying for anything that didn't give him a gain and include Dudley or Petunia, he was greedy.
Chapter 4, pg. 59 The Keeper of the Keys
"I AM NOT PAYING FOR SOME CRACKPOT OLD FOOL TO TEACH HIM MAGIC TRICKS"
Also, the above quote about Harry stealing the house. That situation is something that Vernon would do.
And they got over the fact that he had to go school really fast. After leaving him at platform 9 3/4 Vernon says:
"Have a good term," said Uncle Vernon with an even nastier smile. He
left without another word. Harry turned and saw the Dursleys drive
What Vernon says goes, in most cases.
Chapter 3, The Dursleys Departing, Deathly Hallows
"Don't you take that tone-" began Aunt Petunia in a shrill voice, but
Vernon Dursley waved her down."
And similar events like this happened as well.
And Dudley was just following his parents
Chapter 3, Deathly Hallows, The Dursleys Departing
It seemed that Dudley was struggling with concepts too difficult to
put into words. After several moments of apparently painful internal
struggle he said, "But where's he going to go?" ...
"I don't think you're a waste of space."
Again, Dudley appeared to grapple with thoughts too unwieldy for
expression before mumbling, "You saved my life."
He didn't hate Harry and would have probably befriended him if his parents had not abused Harry.
This goes to a quote:
"You meet the parent when you meet the child."
You might not care for this source. But they had a interesting point of view.
However, they disliked him and treated him badly, while spoiling
When Harry was sent a letter from Hogwarts School of Witchcraft and
Wizardry, Vernon got very angry, as he detested anything to do with
"There's no such thing as magic!" —Vernon trying to convince Harry
Potter to forget about the wizarding world.