I believe it's implied that it wasn't a logical or conscious realization; it's a 'link' issue; being her family, she had a bond of some sort with them, and was able to realize that none of the proffered pigs were them.
Kind of consistent with a lot of moral stuff both in the movie and in the genre in general; if you notice, she didn't solve many problems in the movie with brainpower, so much as with 'heart', courage, or determination. She was able to identify (or fail to, in this case) her parents because they were her parents, and there was a significant bond; had there not been one, she would have failed, and deserved to fail, as they weren't significantly a family. (Slightly heavy-handed for a moral point, but it was a kids movie :) )
Edit: See Oriol's answer for quotes from the author supporting the logic.
A relevant chunk: "...Chihiro simply knows that her parents aren’t there. You ask why she knows, but knowing is human life. That’s all it is."