I finally found an answer from Martin himself:
BHW: Why is the wall so tall?
Martin: To keep out bad things.
Hadrian's Wall was an inspiration. In fantasy, everything is bigger. A
friend of mine, Lisa Tuttle (we wrote Windhaven together), had just
moved to Scotland and was giving me a tour. We were driving in her car
and got to Hadrian's Wall at the end of the day. The tour buses were
leaving. We walked along the top of the wall just as the sun was going
down. It was the fall. I stood there and looked out over the hills of
Scotland and wondered what it would be like to be a Roman centurion
from Italy , Greece , or even Africa , covered in furs and not knowing
what would be coming out of the north at you. I wanted to capture that
feeling.
Hadrian's Wall is impressive, but it's not really tall. A good ladder
would be all you need to scramble right on over it. When you're doing
fantasy, it has to be bigger than in real life. The castles are
grander. Fantasy is painted in larger scale and brighter colors.
Source: George R.R. Martin Talks Ice and Fire, A Book Help Web Exclusive Interview (Archived: 2007, 2014)
So it seems the answer is not much more than "because that's how things are in fantasy books"!