He comes out of no where and I haven't read the books. He just keeps torturing him and giving hints on who he is...
Answer in spoiler:
Roose Bolton's bastard, Ramsay. Their sigil is the flayed man.
Spoiler alert :
Roose Bolton's bastard : Ramsay Snow. Why? - He's not a very nice man.
The other answers have given spoilers on his name. I'll give an explanation of his motivation without giving away too much (quoting myself here)
He's basically a psychopath. He loves torturing people. At one point he married a highborn lady for her inheritance of land. He then locked her up in a tower with no food. She starved to death -- but not before eating off a few of her own fingers.
He also captures girls, releases them naked into the woods, and hunts them. He rapes, kills, and flays them when caught.
He loves doing these things; it gives him great pleasure to watch others suffer (he's a sadist).
All this aside, there is a reason why he is doing this to Theon. He wants Theon to lose almost all sense of identity and remain as his meek servant. He also has some plans that will require Theon's help to be executed well.
Roose Bolton's bastard son Ramsay Snow. Remember, Roose told Robb that he could send his bastard son to recapture Winterfall? And Robb said OK, but he wanted Theon taken alive. So Ramsay captured Theon and took him to Bolton castle at Dreadfort. The Bolton's sigil is the 'Flayed Man' because they are proud to be known for torturing their enemies.
So there you have it.