Alright, so the Dursleys are not poor at all. Vernon Dursley brings home enough money for them to live in an affluent, middle-class area and buy their spoilt son a fair share of the 37 presents (plus 2 more) he receives for his eleventh birthday. Now, like many people in this situation, I'm sure they'd have been happy to have more money, in fact, Harry himself is afraid that they'd try and take it from him!
The Dursleys couldn't have known about this or they'd have had it from him faster than blinking. How often had they complained how much Harry cost them to keep? And all the time there had been a small fortune belonging to him, buried deep under London.
Harry Potter and the Philosopher's Stone - p.58 - Bloomsbury - Chapter 5, Diagon Alley
But this, to me, is a reflection of their greed. Petunia Dursley emphatically does not need hated Harry to give her money.
I'm gonna take it you've not read Harry Potter and the Deathly Hallows, so beware the next bit and I'll refrain from providing quotes:
She loved her sister and she resents the magical world, because of what she feels she lost to it. She also envies her sister and her abilities and she grows to resent her sister because of that and because she feels abandoned by her, left behind, and - I suppose - common.
Because of this, she resents Harry and everything that he is and the world he truly belongs to.
The Dursleys are also a frankly superb metaphor for snobbery and, frankly, bigotry. They do not like what is different from them and they do not want the world to know what's lurking in their family tree:
If it did ... if it got out that they were related to a pair of - well, he didn't think he could bear it.
Harry Potter and the Philosopher's Stone - p.11 - Bloomsbury - Chapter 1, The Boy Who Lived
'Knew!' shrieked Aunt Petunia suddenly. 'Knew! Of course we knew! How could not be, my dratted sister being what she was? Oh, she got a letter just like that and disappeared off to that - that school - and came home every holiday with her pockets full of frog-spawn, turning teacups into rats. I was the only one who saw her for what she was - a freak! But for my mother and father, oh no, it was Lily this and Lily that, they were proud of having a witch in the family!'
Harry Potter and the Philosopher's Stone - p.44 - Bloomsbury - Chapter 4, The Keeper of the Keys
They also make the old mistake. They believe - as many throughout history have done - that if they keep him downtrodden and wretched, they can fix him. As Vernon Dursley says:
'Now, you listen here boy,' he snarled. 'I accept there's something strange about you, probably nothing a good beating wouldn't have cured - and as for all this about your parents, well, they were weirdos, no denying it, and the world's better off without them in my opinion - asked for all they got, getting mixed up with these wizarding types - just what I expected, always knew they'd come to a sticky end -'
Harry Potter and the Philosopher's Stone - p.46 - Bloomsbury - Chapter 4, The Keeper of the Keys
I also think you make a bit of a mistake. Harry Potter is not (in my analysis, anyway) that wealthy. For one thing, he is an orphan, which means that the money he does inherit is not being replenished by a working family as he draws from it. Harry Potter had enough money in his vault to take money off the table. He could go through seven years at school, without - obviously - being able to earn a penny, in the wizarding world, without needing to worry about money. But he wasn't super-wealthy.
Once Harry had refilled his money bag with gold Galleons, silver Sickles and bronze Knuts from his vault at Gringotts, he needed to exercise a lot of self-control not to spend the whole lot at once. He had to keep reminding himself that he had five years to go at Hogwarts, and how it would feel to ask the Dursleys for money for spellbooks ...
Prince on request ... Harry didn't like to think how much gold the Firebolt would cost. He had never wanted anything so much in his life - but ... what was the point in emptying his Gringotts vault for the Firebolt ...?
Harry Potter and the Prisoner of Azkaban - pp.43-4 - Bloomsbury - Chapter 4, The Leaky Cauldron
(Redacted to remove minor spoilers from the first two books)
(Spoiler for Harry Potter and the Order of the Phoenix):
However, his inheritance from Sirius did add to his wealth - and the Dursleys showed an interest - and who can blame Harry for not wanting to share it with them. And frankly, he didn't need the bargaining chip at this stage.