Malfoy was derisive of Hufflepuff House before he even arrived at Hogwarts:
"Well, no one really knows until they get there, do they, but I know I'll be in Slytherin, all our family have been — imagine being in Hufflepuff, I think I'd leave, wouldn't you?"
-- Draco Malfoy, HP and the Philosopher's Stone
But then he's never been known for his acceptance of non-Slytherins in any case.
The Sorting Hat's song (or one of them) suggests Hufflepuff is the house for leftovers, those who aren't considered 'good enough' (by various definitions of 'good') to fit into any other:
Said Slytherin, "We'll teach just those whose ancestry's purest."
Said Ravenclaw, "We'll teach those whose intelligence is surest"
Said Gryffindor, "We'll teach all those with brave deeds to their name."
Said Hufflepuff, "I'll teach the lot and treat them just the same."
For instance, Slytherin took only pure-blood wizards of great cunning just like him.
And only those of sharpest mind were taught by Ravenclaw
While the bravest and the boldest went to daring Gryffindor.
Good Hufflepuff, she took the rest and taught them all she knew,
-- Sorting Hat, HP and the Order of the Phoenix
There's also a line to suggest Hufflepuff is generally underappreciated:
Cedric was walking away from the sort of glory Hufflepuff hadn't had in centuries.
-- HP and the Goblet of Fire (when Cedric Diggory offers to let Harry take the Triwizard Cup)
However, it should be noted that J K Rowling does not believe Hufflepuff is bad:
"This may surprise people, but it is the truth...In many, many ways, Hufflepuff is my favourite House. Here's my reasoning, bear with me. [...] The Hufflepuffs stayed for a different reason; they weren't trying to show off, they weren't being reckless, that's the essence of Hufflepuff House. Now my oldest child [...] said to me —and she, by the way, was not sorted into Hufflepuff House— but, she said to me, 'I think we should all want to be Hufflepuffs.' I can only say to you, that I would not be at all disappointed to be sorted into Hufflepuff House."
-- J K Rowling, web interview Beyond the Page