Dumbledore cared, but Harrys protection trumped a happy life.
Dumbledore felt massive regret for having left Harry with the Dursley's, but the protection they provided trumped all. Below are some big chunks of the book, but they really explain fully why Dumbledore accepted the mistreatment for the greater good.
‘Five years ago you arrived at Hogwarts, Harry, safe and whole, as I
had planned and intended. Well – not quite whole. You had suffered. I
knew you would when I left you on your aunt and uncle’s doorstep. I
knew I was condemning you to ten dark and difficult years.’
‘You might ask – and with good reason – why it had to be so. Why could
some wizarding family not have taken you in? Many would have done so
more than gladly, would have been honoured and delighted to raise you
as a son.
‘My answer is that my priority was to keep you alive. You were in more
danger than perhaps anyone but I realised. Voldemort had been
vanquished hours before, but his supporters – and many of them are
almost as terrible as he – were still at large, angry, desperate and
violent. And I had to make my decision, too, with regard to the years
ahead. Did I believe that Voldemort was gone for ever? No. I knew not
whether it would be ten, twenty or fifty years before he returned, but
I was sure he would do so, and I was sure, too, knowing him as I have
done, that he would not rest until he killed you.
‘I knew that Voldemort’s knowledge of magic is perhaps more extensive
than any wizard alive. I knew that even my most complex and powerful
protective spells and charms were unlikely to be invincible if he ever
returned to full power.
‘But I knew, too, where Voldemort was weak. And so I made my decision.
You would be protected by an ancient magic of which he knows, which he
despises, and which he has always, therefore, underestimated – to his
cost. I am speaking, of course, of the fact that your mother died to
save you. She gave you a lingering protection he never expected, a
protection that flows in your veins to this day. I put my trust,
therefore, in your mother’s blood. I delivered you to her sister, her
only remaining relative.’
‘While you can still call home the place where your mother’s blood
dwells, there you cannot be touched or harmed by Voldemort. He shed
her blood, but it lives on in you and her sister. Her blood became
your refuge. You need return there only once a year, but as long as
you can still call it home, whilst you are there he cannot hurt you.
Your aunt knows this. I explained what I had done in the letter I
left, with you, on her doorstep. She knows that allowing you houseroom
may well have kept you alive for the past fifteen years.’
He even yelled at the Dursleys himself, late of course, but he did express his displeasure.
‘You did not do as I asked. You have never treated Harry as
a son. He has known nothing but neglect and often cruelty at
your hands. The best that can be said is that he has at least
escaped the appalling damage you have inflicted upon the
unfortunate boy sitting between you.’
‘The magic I evoked fifteen years ago means that Harry has powerful
protection while he can still call this house home. However miserable
he has been here, however unwelcome, however badly treated, you have
at least, grudgingly, allowed him houseroom. This magic will cease to
operate the moment that Harry turns seventeen; in other words, the
moment he becomes a man. I ask only this: that you allow Harry to
return, once more, to this house, before his seventeenth birthday,
which will ensure that the protection continues until that time.’
As to everyone else, no one but Dumbledore was aware of the treatment Harry received, other then Mrs. Figg, who again was under orders by Dumbledore to keep an eye on Harry but to not really intervene. See Mrs. Figg's assignment here Why did Dumbledore instruct Mrs. Figg to not tell Harry about the Wizarding World, even up to when he was ready to be admitted?
After coming to Hogwarts we have Hagrid, Sirius, Mr. Weasley, and other members of the order all threaten the Dursleys and comment to them about how they are mistreating Harry, and his home life does improve slightly, mostly by just being completely ignored.