Like, if they found him on their doorstep and took him to an orphanage? Would Dumbledore bring him back to them?
They probably didn't have much of a choice.
Ultimately, this situation didn't arise so we can only speculate about what would've happened. However, I think we have every reason to think that the Dursleys were essentially lumbered with Harry, whether they wanted him or not, and that their consent to keep him was relatively superficial.
This is how Dumbledore describes the transaction:
"I delivered you to [Lily's] sister, her only remaining relative.”
“She doesn’t love me,” said Harry at once. “She doesn’t give a damn -”
“But she took you,” Dumbledore cut across him. “She may have taken you grudgingly, furiously, unwillingly, bitterly, yet still she took you, and in doing so, she sealed the charm I placed upon you.
(Order of the Phoenix, Chapter 37, The Lost Prophecy).
Two points are raised by this.
Firstly, Dumbledore is perfectly aware that Petunia took Harry grudgingly and unwillingly. The fact that he left Harry on the doorstep in the middle of the night with a letter rather than seeking a face-to-face meeting reflects this awareness. He was not labouring under the assumption that the Dursleys would be thrilled to take in a surrogate child. He was well aware that the Dursleys' first instinct might be to turn Harry away.
Secondly, Dumbledore says that he had already cast the protective charm on Privet Drive, even before Petunia took Harry in. Petunia merely sealed the charm. This gives the impression that Dumbledore has already arranged matters in advance and that he expects the Dursleys to comply with these arrangements.
Furthermore, Dumbledore kept a close watch on Harry whilst he was staying at the Dursleys.
Harry...is in even greater danger now than the day when I left him upon your doorstep fifteen years ago, with a letter explaining about his parents’ murder and expressing the hope that you would care for him as though he were your own...You did not do as I asked. You have never treated Harry as a son. He has known nothing but neglect and often cruelty at your hands. The best that can be said is that he has at least escaped the appalling damage you have inflicted upon the unfortunate boy sitting between you.”
(Half-Blood Prince, Chapter 3, Will and Won't).
Dumbledore knows the sort of life Harry has had at Privet Drive, including the cruel treatment he received from the Dursleys. He could've gleaned this information by spying on Harry or simply have picked it up from the conversations he'd had with Harry at Hogwarts. However, remember that he had a tailor-made Order informant on Harry's doorstep in Mrs Figg. Regardless of which source he used, Dumbledore was certainly aware of the harsh reality of Harry's life at the Dursleys. He would certainly have known if they had shipped Harry off to an orphanage.
Think about how that situation plays out for the Dursleys. They have two main fears - magic and scandal - that mutually feed one another. On day one they have an unwanted baby. They send him to an orphanage. On day two they have that same baby back except now he is accompanied by a pissed off Dumbledore. Remember that the Dursleys are paranoid.
The Dursleys had everything they wanted, but they also had a secret, and their greatest fear was that somebody would discover it. They didn't think they could bear it if anyone found out about the Potters...The Dursleys shuddered to think what the neighbours would say if the Potters arrived in the street.
(Philosopher's Stone, Chapter 1, The Boy Who Lived).
So the Dursleys fear the scandal that the Potters could bring upon their reputation. Harry is clearly unwelcome in that regard. But imagine their horror if a fully fledged adult wizard arrived in the street in broad daylight to accompany him. They would have been terrified of such an open confrontation. To their minds, the best way to resolve the situation was to take Harry but raise him as a Muggle, with the hope of crushing the magic out of him.
As Gallifreyan says, Dumbledore's letter probably made use of forceful persuasion to make its point. Here are some things that Dumbledore could have said in it:
- Lily and James are dead. The letter was the first that Petunia has heard of her sister's brutal murder. Dumbledore can count on the sympathy factor.
- You are Harry's only living relative. Harry has nowhere else to go and no other relative to look after him.
- Harry is in real danger. Only the Privet Drive charm can protect him should Voldemort return.
- The Charm has already been performed. As per above, Dumbledore has already arranged things and simply expects the Dursleys to play along.
- If you don't take Harry then the Ministry of Magic may force you to anyway. Who knows what the Ministry protocol was for orphaned magic children? They may well have taken them to the next of kin.
- You once wished to enter the wizarding world yourself. Dumbledore can remind Petunia of the last time they communed by letter and of how willing Petunia was to be a witch on that occasion. She may have hardened her views of magic over the years but Dumbledore may have argued that taking Harry would be Petunia's way of honouring that childhood desire.