Let's keep aside the fact that Aunt Petunia was scared of wizards and overall magic because of her past with Lily. I don't find this a valid reason for bullying Harry or treating him like a lowly servant. To avoid Harry's contact with magic, Vernon Dursley had convinced him that magic didn't exist and it seemed to work.
If we use the logic behind this question, How much does a Horcrux affect its surroundings, people in its immediate vicinity?
Horcrux affects the person who possesses it in a negative manner. Other Horcruxes were inanimate (except Nagini) so it was easy to possess them, wear the ring, read the diary and so on. But in context with Harry, Dursleys were officially declared his guardians, thus making Harry a possession of the Dursleys.
Was it because Harry was a Horcrux, the Dursleys were always mean? I mean their antics were not limited to Harry.
They were over-pampering Dudley, thus making him useless in the real world.
Dudley is shown mistreating the snake at zoo. Thus, we can assume that he mistreated other animals too.
Also, the Dementors attack Dudley as if he was in possession of a Horcrux or to say a criminal soul, thus attracting attention of Dementors.
Had Dursleys been so mean from inception itself, they would have put Harry in an orphanage, which means they must have been good people before. It must have been the prolonged exposure to Horcrux (Harry Potter) that made them act mean.
(It can also be pointed out that Ron also stays around with Harry for long, so why doesn't it affect him. I'd say he is around him but doesn't posses him in any manner. Also, Ron tends to break contact with him regularly when sleeping in different beds, staying away during vacations and so on thus avoid the full effect on himself.)
Any confirmation on this?