I can think of a lot of reasons.

1. Chamber of Secrets reveals (well, technically the very end of Philosopher's Stone reveals) that the Dursleys are unaware that you aren't allowed to do magic outside of Hogwarts at Harry's age. So as far as they're concerned, the boy they abused for 11 years is going to stay in their house for three months out of the year with the ability to torment them.
2. The Dursleys are shown to be absolutely obsessed with normalcy and with what their neighbors think of them. They live in constant fear that the neighbors will discover that Harry is magic, which of course would be exasperated by him actually learning magic.
3. They really, really hate magic. Like, a lot.
4. They really, really hate Harry. When he asks permission to go the Quidditich World Cup, the narration describes that Uncle Dursley is torn. He'd really like Harry out of his hair for a month... but it would also have the side effect of making Harry happy.
5. For Petunia, I think it's a straightforward case of jealousy. She always wanted to be a witch and go to Hogwarts, and she simply despises Lily and Harry and Severus for being able to go while she can't.

I can think of many reasons.

1. Chamber of Secrets reveals (well, technically the very end of Philosopher's Stone reveals) that the Dursleys are unaware that you aren't allowed to do magic outside of Hogwarts at Harry's age. So as far as they're concerned, the boy they abused for 11 years is going to stay in their house for three months out of the year with the ability to torment them.
2. The Dursleys are shown to be absolutely obsessed with normalcy and with what their neighbors think of them. They live in constant fear that the neighbors will discover that Harry is magic, which of course would be exacerbated by him actually learning magic.
3. They really, really hate magic. Like, a lot.
4. They really, really hate Harry. When he asks permission to go the Quidditich World Cup, the narration describes that Uncle Dursley is torn. He'd really like Harry out of his hair for a month... but it would also have the side effect of making Harry happy.
5. For Petunia, I think it's a straightforward case of jealousy. She always wanted to be a witch and go to Hogwarts, and she simply despises Lily and Harry and Severus for being able to go while she can't.