I think that one would not always be able to anticipate how a spell will affect a non-human (or a human, for that matter). There seem to be examples of magic working as anticipated, as well as not as anticipated, in canon.
In Goblet of Fire, Barty Crouch Jr. as Moody demonstrated that the Unforgivables work on spiders in exactly the same way they do with humans -- the spider was forced to act against its will, it was painfully tortured, and it was killed. As well, Crouch!Moody teaches Harry's class how to resist the Imperius Curse and some are able to do this better than others.)
At the beginning of HBP, in the chapter Spinner's End, Bellatrix Lestrange kills a fox with what appears to be Avada Kedavra (green jet of light, but no audible incantation; however, as she states she thought the fox might be an Auror, Avada Kedavra seems the logical curse).
In OOTP, Hagrid withstands Stunning spells coming from six individuals at once, due to the giant in him (conversely, Professor McGonagall survives four Stunning spells to the chest at once, which apparently could have been lethal for someone of her age and stature, so sometimes humans don't react to spells in the way they're supposed to.) Umbridge's Incarcerous spell works as it should on Magorian the Centaur in OOTP.
Voldemort did not anticipate that Kreacher the house-elf would not be killed by drinking the locket Horcrux potion, nor did he imagine that Kreacher -- whose magic is different than wizarding magic -- would be able to Apparate from within the Inferi lake back to No. 12 Grimmauld Place. Voldemort kills a Gringotts goblin with Avada Kedavra in DH.
Sirius Black and Remus Lupin were able to Untransfigure Wormtail back into his human form from his rat body as would be expected in PoA.
Fantastic Beasts and Where to Find Them notes the magical creatures that do not react readily to all magic (spells charms, and curses): Chimaeras; Yeti; Quintapeds; Erumpents; Nudus; Kappas; Lethifolds; Manticores; and Trolls.