The answer to this question is a bit complicated. As the quote provided by rand al'thor states, Harry will be under a protection spell as long as he calls Privet Drive his home. The quote also states Harry could not be touched by Voldemort, although he is touched be Voldemort multiple times before he turns 17 (the protection wears off after he is 17). Maybe, the protection spell began to wear off because Harry didn't really consider Privet Drive his home anymore after arriving at Hogwarts (emphasis mine).
Perhaps it was because he was now so busy, what with Quidditch
practice three evenings a week on top of all his homework, but
Harry could hardly believe it when he realized that he’d already
been at Hogwarts two months. The castle felt more like home than
Privet Drive ever had.
If the protection spell was truly beginning to wear off after Harry's first year, there was no point in him returning to Privet Drive every year. Let's look at the evidence, and compare Harry touching Quirrell in SS/PS to Voldemort touching Harry in GoF (emphasis mine).
Harry felt Quirrell’s hand
close on his wrist. At once, a needle-sharp pain seared across
Harry’s scar; his head felt as though it was about to split in two; he
yelled, struggling with all his might, and to his surprise, Quirrell let
go of him. The pain in his head lessened — he looked around
wildly to see where Quirrell had gone, and saw him hunched in
pain, looking at his fingers — they were blistering before his eyes . . . “Master, I cannot hold him — my hands — my hands!”
Quirrell can't touch Harry without screaming in pain.
know that on the night I lost my powers and my body, I tried to kill
him. His mother died in the attempt to save him — and unwittingly
provided him with a protection I admit I had not foreseen.
. . . I could not touch the boy.”
Voldemort raised one of his long white fingers and put it very
close to Harry’s cheek.
“His mother left upon him the traces of her sacrifice. . . . This is
old magic, I should have remembered it, I was foolish to overlook
it . . . but no matter. I can touch him now.”
Harry felt the cold tip of the long white finger touch him, and
thought his head would burst with the pain. Voldemort laughed
softly in his ear, then took the finger away and continued addressing
the Death Eaters.
Not only does Voldemort not scream, but he laughs. This could only mean the protection had worn off at some point between the first year and fourth year, and Harry was free to live with the Weasleys. He kept coming back to Privet Drive afterwards for plot reasons only.