In Game of Thrones S03E10, immediately after learning that...
Robb Stark has died,
...the following conversation takes place between Tywin Lannister and Tyrion Lannister:
Tyrion: ...it's easy for you to preach utter devotion to family when you're making all the decisions.
Tywin: Easy for me, is it?
Tyrion: When have you ever done something that wasn't in your interest but solely for the benefit of the family?
Tywin: The day that you were born. I wanted to carry you into the sea and let the waves wash you away. Instead, I let you live. And I brought you as my son. Because you're a Lannister.
Tyrion believes that his father's hatred for him is because he is a dwarf. This is evident from this statement from him to Jon Snow:
All dwarfs are bastards in their father's eyes.
However, dwarfism is not noticeable until the later years of childhood. There is no way Tywin could have known that Tyrion was going to be a dwarf. This is confirmed by this statement from Oberyn Martel from S04E07:
When we met your sister, she promised she would show you to us. Every day we would ask. Every day she would say, "Soon." Then she and your brother took us to your nursery and... she unveiled the freak. Your head was a bit large. Your arms and legs were a bit small, but no claw. No red eye. No tail between your legs. Just a tiny pink cock. We didn't try to hide our disappointment. "That's not a monster," I told Cersei, "that's just a baby." ...
So this confirms that although Tyrion had minor deformities on birth, his physical appearance was mostly normal. That's not a reason to kill your own child, even by GoT universe standards.
So what caused Tywin Lannister to hate Tyrion so much right from his birth?