Aside from a clash of personalities, the most obvious case is that Rubeus Hagrid was suspected of originally opening the Chamber of Secrets, which lead to the death of a student and Hagrid's public expulsion from Hogwarts.
Although Hagrid was framed, it wasn't until after the events in Chamber of Secrets that the actions of the Trio were enough to acquit Hagrid of accusations related to the first opening of the Chamber of Secrets (actually performed by Riddle).
Filch is far and away a stickler for rules, even when Dumbledore may not agree with with them:
‘Mr Filch, the caretaker, has asked me to tell you that the list of objects forbidden inside the castle has this year been extended to include Screaming Yo-yos, Fanged Frisbees and Ever-Bashing Boomerangs. The full list comprises some four hundred and thirty-seven items, I believe, and can be viewed in Mr Filch’s office, if anybody would like to check it.’
The corners of Dumbledore’s mouth twitched.
-Goblet of Fire
and has a penchant for people being punished:
‘I bet you’ll think twice about breaking a school rule again, won’t you, eh?’ he continued, leering at them. ‘Oh yes ... hard work and pain are the best teachers if you ask me ... It’s just a pity they let the old punishments die out ... hang you by your wrists from the ceiling for a few days, I’ve got the chains still in my office, keep ’em well oiled in case they’re ever needed ... Right, off we go, and don’t think of running off, now, it’ll be worse for you if you do.’
-Philosopher's Stone/Sorcerer's Stone
Furthermore, we see that he flourished when Umbridge and even Death Eaters took over Hogwarts with their excessive rules and cruel punishments. To him, Hagrid constantly broke the rules and essentially got away with it.
On top of that, Hagrid is a bit of a bumbling man. He's a half-giant, which is abnormal. He hides a broken wand in his umbrella, which we don't know if Filch suspects is going on. And, Hagrid has a habit of handling dangerous or illegal animals, or even possibly violating the Ban on Experimental Breeding as seen with the Blast-Ended Skrewts. We can't know for sure if they were pre-sanctioned for the Triwizard Tournament, as he was never charged. Rita's article seems to suggest he didn't have permission, but then again Arthur seemed to dabble in a lot of Misuse of Muggle Artifacts. Nevertheless, even if he had 100% permission from the beginning, it furthers his appearance as someone who gets to to break the rules. Who else but Dumbledore's favorite magical animal keeper would be allowed an exception to a major Wizarding law? Incidents like that would likely gall someone such as Filch.
In general, Hagrid has the appearance of a chronic rule-breaker that gets away with it because he's the headmaster's favorite, which seems to be precisely the type of reason for Filch to dislike Hagrid, and thus to have Mrs. Norris spy on him. If Mrs. Norris can catch Hagrid in the act of breaking a major rule, then I'm sure Filch thinks Dumbledore will have no choice but to punish or axe Hagrid.