You are correct. Non-spell magic would not alert the Ministry. This is essentially explicitly stated by Moody a few lines later (my emphasis):
"We're going to use the only means of transport left to us, the only
ones the Trace can't detect, because we don't need to cast spells to
use them: brooms, thestrals, and Hagrid's motorbike."
Using potions would (generally speaking) not involve casting any spells, and indeed on this very occasion they used the Polyjuice Potion. Actually creating potions might involve the casting of spells at some point, and if such spells were cast the Ministry would indeed be alerted (but that was not the case in this scene). Charms would involve casting spells so they would be detected by the Ministry.