A Song of Ice and Fire
In the books he definitely knew his mother after having her and Reek 1 teach him about his heritage and make him aspire to take the Bolton name.
"Smitten?" Bolton laughed. "Did he use that word? Why, the boy has a singer's soul … though if you believe that song, you may well be dimmer than the first Reek. Even the riding part is wrong. I was hunting a fox along the Weeping Water when I chanced upon a mill and saw a young woman washing clothes in the stream. The old miller had gotten himself a new young wife, a girl not half his age. She was a tall, willowy creature, very healthy-looking. Long legs and small firm breasts, like two ripe plums. Pretty, in a common sort of way. The moment that I set eyes on her I wanted her. Such was my due. The maesters will tell you that King Jaehaerys abolished the lord's right to the first night to appease his shrewish queen, but where the old gods rule, old customs linger. The Umbers keep the first night too, deny it as they may. Certain of the mountain clans as well, and on Skagos … well, only heart trees ever see half of what they do on Skagos.
"This miller's marriage had been performed without my leave or knowledge. The man had cheated me. So I had him hanged, and claimed my rights beneath the tree where he was swaying. If truth be told, the wench was hardly worth the rope. The fox escaped as well, and on our way back to the Dreadfort my favorite courser came up lame, so all in all it was a dismal day.
"A year later this same wench had the impudence to turn up at the Dreadfort with a squalling, red-faced monster that she claimed was my own get. I should've had the mother whipped and thrown her child down a well … but the babe did have my eyes. She told me that when her dead husband's brother saw those eyes, he beat her bloody and drove her from the mill. That annoyed me, so I gave her the mill and had the brother's tongue cut out, to make certain he did not go running to Winterfell with tales that might disturb Lord Rickard. Each year I sent the woman some piglets and chickens and a bag of stars, on the understanding that she was never to tell the boy who had fathered him. A peaceful land, a quiet people, that has always been my rule."
"A fine rule, m'lord."
"The woman disobeyed me, though. You see what Ramsay is. She made him, her and Reek, always whispering in his ear about his rights. He should have been content to grind corn. Does he truly think that he can ever rule the north?"
A Dance with Dragons, Reek III
Game of Thrones
...unless he has some very early memories of her when he was a baby/toddler.
The conversation between Roose and Ramsay seems to indicate that although the origins of Ramsay are the same in the show to the books the events that followed aren't with Ramsay only knowing his mother in the first couple of years of his life. We don't get any deeper into this conversation though as Roose turns towards talking about Stannis instead.
Roose Bolton: You've never asked me about your mother.
Ramsay Snow: Why would I? She had me, she died. And here we are.
Roose Bolton: She was a peasant girl. Pretty in a common sort of way. She was the miller's wife. Apparently they had married without my knowledge or consent. So I had him hanged and I took her beneath the tree where he was swaying. She fought me the whole time. She was lucky I didn't hang her, too. A year later she came to my gates with a squalling baby in her arms. A baby she claimed was mine. I nearly had her whipped and the child thrown in the river.
Game of Thrones, Season 5 Episode 5, "Kill the Boy"