It was most likely the latter; Barty Jr. was acting out on revenge.
Moody was a hard wizard, but still upheld the rules. He was careful not to break any, as he had spent years fighting to uphold them (being an auror). He wouldn't have used transfiguration on a student, no matter how much he hated him. This would've been like stooping to Umbridge's level, which I'm certain Moody would never have done.
There are several references to Moody being a cautious guy. He enforces even the most superfluous of rules like "elementary wand safety".
"Never you mind, you just keep your wand out of your back pocket!" growled Mad-Eye. "Elementary wand safety, nobody bothers about it anymore. . . .”
- Order of The Phoenix, Ch3: "The Advance Guard"
*(I used this specific excerpt because in most others, he is cautious because he's paranoid. But in this specific one, it shows that he cares about rules, even the small ones.)
Even later in that conversation he doesn't let Harry talk about Voldemort there at the Dursley's house. He's very cautious and one who plays by the rules. And after decades of auror-life, has become a very paranoid.
Hard to imagine someone who cares even about "elementary wand safety" would transfigure a student. And hence the conclusion.