They don't really 'look down', as both Men and Elves are children of Ilúvatar.
However they think that humanity is corrupt:
Yet the Elves believe that Men are often a grief to Manwë, who knows most of the mind of Ilúvatar; for it seems to the Elves that Men resemble Melkor most of all the Ainur, although he has ever feared and hated them, even those that served him.
And they may be right, as it seems that Morgoth corruption has been particularly severe on humanity, as:
But the sons of Men die indeed, and leave the world; wherefore they are called the Guests, or the Strangers. Death is their fate, the gift of Ilúvatar, which as Time wears even the Powers shall envy. But Melkor has cast his shadow upon it, and confounded it with darkness, and brought forth evil out of good, and fear out of hope.
Morgoth was the first to find Men and he corrupted them, and according to the legends of Men, he reduced their lifespan and taught them to fear death.
That's their average position, of course you have variations:
The worst: during the Bliss in Valinor, Morgoth spread the lie that Men were going to steal Middle-Earth from the Elves. After the Fall of Numenor, general attitude of contempt toward them (except the line of Elendil).
The best: Great friendship during the First Age, the Elves taught much to the first men, friendship with the Numenorians.