To answer your question, consider the situation.  They had ALREADY send Hulk very far away, to a distant planet.  When he came back, it was as that planet's WARLORD, with an army at his back.

In comics, magic is essentially a tool.  It can do many, many things, but it can be overcome.  Hulk, at this point, is not simply angry.  He's intelligent, focused, and out for vengeance.  Recall - he's come back because (so far as he's aware) Earth and its metahumans have taken virtually everything he cared about from him.  His home, his love, his future, everything.  He is entirely committed to destroying them.

Hulk's strength is great even in normal situations, and increases the angrier he gets.  When he finally lands on Earth, his rage is so great that the comics indicate that he could have shattered the planet with a single stomp.  This strength, since he gained Banner's intellect while in Hulk form, has also reinforced his will.

It's highly likely that any magic that could be resisted by physical strength or willpower would have washed off of Hulk like water off of a duck.

To your specific point about teleportation, please consider the situation: Hulk had an army that didn't have former relationships with Earth's heroes.  If they couldn't beat Hulk physically, their only chance at victory would be to talk him down.

Had they teleported him away, what chance would they have had?  The army would still be there, and would be hungry for blood - their beloved leader would have been seemingly destroyed by their enemies.  They wouldn't have stopped and said, "What did you DO?!"  They would have smashed and conquered.

Moreover, even if they had defeated the army after teleporting Hulk away, what would it gain them?  The last time they sent Hulk away from Earth, look what happened.  Unless they were prepared to kill him (say, by teleporting him into a star or black hole or similar) they'd risk him coming back again, possibly with more resources and ready for their tricks.

If they'd been willing to kill Hulk, they likely would have done it already (say, by having that ship crash into a star).  I believe they are unwilling to kill Hulk (as he has saved their lives many times) and, more practically, uncertain that they COULD if they tried.