Something that I feel everyone forgets is that, other than the blood wards, Dumbledore had other motives for placing Harry with the Dursleys, which he implicitly states. Of course, he probably didn't anticipate they would be as bad as they were- or perhaps he did and sought to take advantage of it. Or maybe I've been reading too much fanfiction.
Anyway. Here's a passage from the Philoshpers Stone:
‘I’ve come to bring Harry to his aunt and uncle. They’re the only
family he has left now.’ ‘You don’t mean – you can’t mean the people who live here ?’ cried Professor McGonagall, jumping to her
pointing at number four. ‘Dumbledore – you can’t. I’ve been watching
them all day. You couldn’t find two people who are less like us. And
they’ve got this son – I saw him kicking his mother all the way up the
street, screaming for sweets. Harry Potter come and live here!’ ‘It’s
the best place for him,’ said Dumbledore firmly. ‘His aunt and uncle
will be able to explain everything to him when he’s older. I’ve
written them a letter.’ ‘A letter?’ repeated Professor McGonagall
faintly, sitting back down on the wall. ‘Really, Dumbledore, you think
you can explain all this in a letter? These people will never
understand him! He’ll be famous – a legend – I wouldn’t be surprised
if today was known as Harry Potter Day in future – there will be books
written about Harry – every child in our world will know his name!’
‘Exactly,’ said Dumbledore, looking very seriously over the top of his
half- moon glasses. ‘It would be enough to turn any boy’s head. Famous
before he can walk and talk! Famous for something he won’t even
remember! Can’t you see how much better off he’ll be, growing up away
from all that until he’s ready to take it?’ Professor McGonagall
opened her mouth, changed her mind, swallowed and then said, ‘Yes –
yes, you’re right, of course. But how is the boy getting here,
Anyway, this infers that Dumbledore somewhat knew that Harry would be mistreated,or, at least, that his placement wouldn't be ideal. I do wonder, though, why Dumbledore hadn't thought so far as to anticipate Harry turning into a second Dudley. But then again, most wizards don't possess even a shred of logic, and Dumbledore, great though he may have been, certainly fits into the category.
But! This begs many questions, like, when Dumbledore saw the letter addressed to the Cupboard Under The Stairs (though the general consensus seems to be that they are written by an automated quill) or that Harry was underweight, why didn't he, or anyone else, for that matter, do anything?
Well, as we see from Harry's point of view, his earlier years, save for the bullying and occasional cuffing by his aunt and uncle, aren't too unbearable. For him, anyway. He loathed going back to the Dursleys, but he was assured that they thought he could cast spells (which is also something I've wondered about; via Lily, Petunia should definitely know about the summer prohibition; she must have cared marginally for Harry) and therefore scare them to avoid any physical abuse.
Plus, even though Harry was small for his age, he was not starved. I haven't the time to find the exact quote, but it goes somewhere along the lines of the Dursleys never "exactly starving" him, and that Dudley would just take anything Harry wanted to eat. Harry was small perhaps because of never eating more than was needed, and running a lot, completing chores, and maybe sleeping in a cupboard somewhat stunted his growth. Since he was only slightly small, and a first year, most probably could ignore that. The wizards, not knowing much of muggles, would most likely think that the Dursleys were horrible muggles, yes, but not much more could be done. I suspect that the term "muggle" after "horrible" made them form a sort of disassociation towards it? And as for Hermione, well, for all her book smarts, they were only eleven at the time.
And so the cycle was cursed to continue. Harry had, in fact, told people that he didn't want to go back, but he was told that he had to. Harry, at that point, had exactly zero trust in adults, McGonagalls stint with ignoring him about the stone withstanding.
Second year; the abuse worsened, and the Weasleys knew. But, what could they do? Let's not fool ourselves into thinking Dumbledore wasn't aware of why and when Harry was "rescued." And but of course, the bars were fixed onto the windows, visible. The neighbors already knew of Harrys criminal status. So Arabella Figg obviously saw the bars, and reported them.
Harry stayed with the Weasleys for the rest of the summer, so relieved and of course heading to Hogwarts, that he forgot.
As much as I'd like to psychoanalyze every single summer, I'm running out of time, so I'll cut it short. Harry was a child, emotionally abused. He didn't trust adults to tell them what awful actually meant; of course they could assume that awful muggles simply meant that they were awful because they lacked magic and Hogwarts was better.
Later, Harry was a rather prideful (as we see from the Blood Quill incidents) teenager. He's used to it, and usually spends the rest of the summer with the Weasleys. And he understands the importance of the Blood Wards.
Dumbledore is tricky. His actions can either waylay him as a manipulative, power hungry old man, wanting to keep his pawn emotionally dependent on the Wizarding World so that he could sacrifice himself "for the Greater Good" when the time was ripe, or he could have been an old man who was tired, and couldn't do much in the grand scheme of things; after all, the blood protections, relating to Lilys sacrifice, were imperative, and he couldn't attack nor simply imperio the muggles. It would be terrible for young Harry, but it was necessary. This Dumbledore wouldn't know of the abuse until it was too late, and by then Voldemort was becoming alarmingly active as the Howgwarts years went by, and Harry was older, and there was nothing to be done.
So!That was my long, uninteresting take! Hope you enjoyed it, I wasted like an hour. Or two. Shit.