It isn't clear if Scabbers being an Animagus would affect the spell.
Transfiguring animals is relatively simple - it's taught to the first-years at Hogwarts. Their Transfiguration exam in their first year required them to turn a mouse into a snuff-box.
“Professor McGonagall watched them turn a mouse into a snuff-box – points were given for how pretty the snuff-box was, but taken away if it had whiskers.” - Harry Potter and the Philosopher's Stone, Chapter 16 (Through the Trapdoor)
Mice are similar creatures to rats, so Transfiguring an average (non-Animagus) rat is presumably also fairly simple.
Transfiguring a human is presumably much more difficult than transfiguring an animal, since Hogwarts students only start learning that in N.E.W.T.-level Transfiguration classes.
“They had just embarked upon the immensely difficult topic of human transfiguration; working in front of mirrors, they were supposed to be changing the colour of their own eyebrows.” - Harry Potter and the Half-Blood Prince, Chapter 15 (The Unbreakable Vow)
Hermione also mentioned how difficult human Transfiguration is when she's helping Harry prepare for the Second Task.
“Of course, the ideal solution would be for you to Transfigure yourself into a submarine or something,’ she said. ‘If only we’d done human Transfiguration already! But I don’t think we start that until sixth year, and it can go badly wrong if you don’t know what you’re doing …” - Harry Potter and the Goblet of Fire, Chapter 26 (Yhe Second Task)
However, it's never mentioned (or tried in any situation with no other variables) whether transfiguring an Animagus would be like transfiguring a human, an animal, something between the two, or if it would even work at all. Transfiguring an Animagus isn't quite like transfiguring a human, and isn't quite like transfiguring an animal, so it's not clear what would happen.
In the book, his second-year spellcasting problems are because his wand is broken.
In the book, Ron never tries to transfigure Scabbers into a goblet. However, all his spells in his second year failed, ever since his wand broke in the encounter with the Whomping Willow.
However, Ron may have still failed, even with a working wand and a non-Animagus.
Ron's failed Transfiguration was probably not his fault in this case. However, that doesn't mean he would have succeeded given better circumstances. In their first year, Ron doesn't do well at Transfiguration, with a working wand and on animals provided by Professor McGonagall. In later years, he isn't particularly successful at it either.
He doesn't do well at it in his fifth year either. The first time he attempts to Vanish a snail, it maybe gets paled, but that's all.
“She was quite right; Harry found the Vanishing Spells horribly difficult. By the end of a double period neither he nor Ron had managed to vanish the snails on which they were practising, though Ron said hopefully he thought his looked a bit paler.” - Harry Potter and the Order of the Phoenix, Chapter 13 (Detention with Dolores)
He's left with a mouse tail when he tries Vanishing a mouse.
“Well, it’s a start,’ said Ron, holding up a long wriggling mouse-tail and dropping it back into the box Lavender was passing around.” - Harry Potter and the Order of the Phoenix, Chapter 15 (The Hogwarts High Inquisitor)