J.K. Rowling said Harry wasn’t contaminated by the piece of soul.
When addressing whether Harry is a true Horcrux or not, J.K. Rowling said that Harry himself wasn’t contaminated by the bit of soul. However, she also says that the piece of soul fed off of his negative emotions during his fifth year, and stirring inside him because of that.
JKR: And so this part of it flies off, and attaches to the only living thing in the room. A part of it flees in the very-close-to-death limbo state that Voldemort then goes on and exists in. I suppose it’s very close to being a Horcrux, but Harry did not become an evil object. He didn’t have curses upon him that the other Horcruxes had. He himself was not contaminated by carrying this bit of parasitic soul. The only time he ever felt it stirring and moving was in Order of the Phoenix, when he himself goes through a very dark time. And there’s a moment where he’s looking at Dumbledore, and he feels something rear like a snake inside him, and of course, at those times, it’s because the piece of soul inside him is feeding off his emotions. He’s going through a dark time, and that piece of soul is enjoying it, and making its presence felt, but he doesn’t know what he’s feeling, of course. Also, I always imagine that the Sorting Hat detected the presence of that piece of soul (JN: Yeah!) when Harry first tried it on, because it’s strongly tempted to put him in Slytherin. So that’s how I see it.
- PotterCast (Dec 23, 2007)
He also did feel the pain of the soul trying to rejoin its master when the Dark Lord was particularly active.
JKR: Well, of course the pain he feels whenever Voldemort’s particularly active is this piece of soul seeking to rejoin the master soul. When his scar is hurting him so much, that’s not scar tissue hurting him. That’s this piece of soul really wanting to get back out the way it entered. It really wants to- it entered this boy’s body through a wound, and it wants to rejoin the master. So when Voldemort’s near him, when he’s particularly active, this connection, (JN: Oh, my gosh!) it was always there. That’s what I always imagined this pain was. Yes, so there you go.
- PotterCast (Dec 23, 2007)
During the years when the Dark Lord wasn’t active, and was far away in spirit form, too weak to do anything, Harry didn’t really feel any effects from the soul piece other than his unexpected ability to speak Parseltongue. It seems likely, then, if the Dark Lord never was resurrected after his attack on the Potters and remained both weak and far away from Harry, that Harry wouldn’t feel any effects from the soul piece inside him.
The pain is caused by the piece of soul trying to rejoin the master soul, but it certainly seems to worsen when the Dark Lord is actively plotting against Harry, so it’s certainly possible - but not confirmed - that if he decided not to bother with hunting down Harry, that the piece of soul within Harry wouldn’t hurt as much.
Of course, the piece of soul is killed, and Harry presumably had no effects other than losing the link to Voldemort and the powers that came with it like Parseltongue.