Dudley was incredibly spoilt by his parents when he was a child. He was brought up very badly, and he’s picked up all of their worst habits. By the end of the fourth book, he’s selfish, obese, rude, insolent, a bully, and taught to sneer at anybody who isn’t like him. A terrible preparation to go out into the world. That’s the “damage” Dumbledore is referring to.
In an interview in 2000, JK Rowling expressed some sympathy for the way Dudley was raised:
“I like torturing them,” said Rowling. “You should keep an eye on Dudley. It’s probably too late for Aunt Petunia and Uncle Vernon. I feel sorry for Dudley. I might joke about him, but I feel truly sorry for him because I see him as just as abused as Harry. Though, in possibly a less obvious way. What they are doing to him is inept, really. I think children recognize that. Poor Dudley. He’s not being prepared for the world at all, in any reasonable or compassionate way, so I feel sorry for him. But there’s something funny about him, also. The pig’s tail was irresistible.”
This point was emphasised again in an interview at a 2007 book event, when she explained what Dudley saw during his Dementor attack in Order of the Phoenix:
“People usually ask me, what is it that Dudley saw during the Dementor attack?” Rowling said. “My feeling is that he saw himself, exactly for what he was, and for a boy that spoiled, it would be terrifying. So he was jolted out of it. Dementor attacks aren’t usually good for people, but this one was.”
We see him mellow in the later books, as he realises the damage done to him so far, and tries to improve himself.