The things she did as an Animagus were debatably legal...
The thing about Rita Skeeter is she always goes for the Best stories possible, like when she sees Harry faint in Divination class, but Dumbledore had banned her from school grounds and if he had known she was an Animagus he would of used extra measures to keep her out.
"I found out how she was listening in on private conversations when she wasn't supposed to be coming onto the grounds," said Hermione in a rush.
- Harry Potter and the Goblet of Fire, Chapter 37 (The Beginning)
Also, i doubt that just if you're a Animagus you're allowed to creep around and spy/listen in on private conversations.
"There was a beetle on the statue the night we heard Hagrid telling Madame Maxime about his mum!" "Exactly," said Hermione. "And Viktor pulled a beetle out of my hair after we'd had our conversation by the lake. And unless I'm very much mistaken, Rita was perched on the windowsill of the Divinaion class the day your scar hurt. She's been buzzing around for stories all year."
- Harry Potter and the Goblet of Fire, Chapter 37 (The Beginning)
So her stories would lose their exclusivity if she registered. Her reputation would be lost and she might lose her spot as a Reporter for the Daily Prophet because she would then be publicly viewed as a sneak.
The sensationalist Daily Prophet reporter whom Dumbledore had banned from Hogwarts grounds managed somehow to wheedle her way back in by changing into her illegal Animagus form: that of a nasty little beetle. Appropriate enough for a journalist who scuttled merrily over the misfortune of others and fed on trash, you might say.
- Unregistered Animagi - Pottermore
Also i don't think that she figured out Hermione couldn't actually prove it, I think Rita thought that Hermione's word would be enough to convict her.
The punishment for being an unregistered Animagus was time in Azkaban so, when clever-clogs Hermione discovered Rita’s secret, the threat of exposure was enough leverage to make her stop writing those poisonous articles.
- Unregistered Animagi - Pottermore
Finally, Hermione's story makes sense, and there are people who would back it up, as @Holger mentioned she has probably used her ability as an Animagus to get good stories before, and people would wonder how she got those stories and if Hermione revealed to the public that Rita was an Animagus the public would think that her story made sense and support her case, eventually leading to the conviction of Rita.