Randyll Tarly hasn't been very heavily featured in the books as of yet, other than references Sam has made, therefore there is little canonical information about him. However, we can speculate based on what we do know.
We know that he places honor very highly on his list of priorities. He staunchly supported Aerys Targaryen, his rightful king, during Robert's rebellion, as well as supporting his liege lord Mace Tyrell during the War of the Five Kings.
He also didn't want shame brought upon his house by Sam inheriting Horn Hill, as he saw him as weak as he wasn't good at fighting and wouldn't have been able to lead his people into battle.
He also didn't want Sam to become a Maester, despite the fact that he was clearly very well suited to the role, as he didn't want his son serving other lords. Even though he despises his son, he still sees that he has the Tarly name and wants him to do what he can to bring pride to it.
So considering that he places so much pride in his name and his honor, simply having the perception that Wildlings are bad people, and don't belong in Westeros (as most people in the seven kingdoms believe), would be enough for him to form a very strong opinion as to what is right and wrong with something he realistically knows absolutely nothing about, simply because the realm at large considers them an enemy.
He may also have been angry that his oldest son had not only broken his oath to the Night's Watch to father no children, he had the child with an enemy of the kingdoms, and if word got out it would bring a lot of dishonor to him and his family.