They did indeed see them on their way down, but they were in such a hurry (attempting to remain on the planet for as short a time as possible) that they didn't realise that the "mountains" were moving, nor have time to conduct even a cursory examination of the surroundings.
Amelia looked in the direction the robot indicated. The water
stretched out to the horizon, where it met a mountain range, misty
with distance; one long ridge that vanished in each direction. The
sight of the alien skyline arrested her for a moment, and she wished
they weren’t in such a hurry. She had long dreamt of her first moments
on an extra-solar planet, and this wasn’t how it was supposed to go.
There should be a little ceremony, a little “That’s one small step.”
Instead they were in this tearing hurry, and it felt completely
half-assed. But it was what it was. They weren’t here to set up flags
and take pictures.
Interstellar: A Novel
As far as what they could see from orbit, in the movie the waves look an awful lot like mountain ranges.
The script also indicates that at the point at which they could have probably spotted that the mountains were fluid, they passed through a cloud layer and were otherwise distracted.
EXT. MILLER’S PLANET - CONTINUOUS
The Ranger CUTS through cloud formations, BURSTING out into
CLEARER AIR, HIGH ABOVE AN ENDLESS OCEAN -