They gave it to him in the first book, after the incident with the letters.. but as to keeping it, I don't remember it being expressly stated, but I believe it was due to them being afraid of what he might do with his magic, once he learned he had it, and how to use it.
From the same book:
Underage wizards weren’t allowed to use magic outside of school. Harry hadn’t told the Dursleys this; he knew it was only their terror that he might turn them all into dung beetles that stopped them from locking him in the cupboard under the stairs with his wand and broomstick.
Since that was where he stayed before, it seems pretty consistent with that being the reason.
In the first book, near the end, we see:
“Hope you have—er—a good holiday,” said Hermione, looking uncertainly after Uncle Vernon, shocked that anyone could be so unpleasant.
“Oh, I will,” said Harry, and they were surprised at the grin that was spreading over his face. “They don’t know we’re not allowed to use magic at home. I’m going to have a lot of fun with Dudley this summer…”
Beyond that, I think they were somewhat afraid of what the Wizardling World might do in response to their 'Hero' being abused.. That didn't result in good treatment, but at least the over-the-top 'keep him under the stairs' issue went away. Overall, a lesser version of the protection he gets at the end of Prisoner.
“It’s not,” said Harry cheerfully. “It’s a letter from my godfather.”
“Godfather?” sputtered Uncle Vernon. “You haven’t got a godfather!”
“Yes, I have,” said Harry brightly. “He was my mum and dad’s best friend. He’s a convicted murderer, but he’s broken out of wizard prison and he’s on the run. He likes to keep in touch with me, though… keep up with my news… check if I’m happy…”
And, grinning broadly at the look of horror on Uncle Vernon’s face, Harry set off toward the station exit, Hedwig rattling along in front of him, for what looked like a much better summer than the last.