The book describes these scene slightly different than the movie, and also contains the explanation (my emphasis):
“You can’t have met many decent wizards,” said Harry, trying to cheer him up.
Dobby shook his head. Then, without warning, he leapt up and started banging his head furiously on the window, shouting, “Bad Dobby! Bad Dobby!”
“Don’t — what are you doing?” Harry hissed, springing up and pulling Dobby back onto the bed — Hedwig had woken up with a particularly loud screech and was beating her wings wildly against the bars of her cage.
“Dobby had to punish himself, sir,” said the elf, who had gone slightly cross-eyed. “Dobby almost spoke ill of his family, sir....”
Thus, the problem with what Dobby said (or in the book communicated nonverbally) is that it implicitly condemned the family that owned him, and/or nearly led to him explicitly condemning them.
