In the movies so far (Thor and Avengers), most of the time, Mjölnir is fairly nearby when Thor calls it to him, so we have no other referent. However, throughout the movies, there seems to be some effort to keep things plausible within the structure of a universe that includes superheroes :-) For example, Thor's abilities are not glossed as magic, nor do the Asgardians, by and large, actually claim to be gods, but rather they're glossed as highly advanced technology (e.g. Clarke's Law) that caused the Vikings they aided against the Frost Giants to think of them as gods.
So, in this case, I think Whedon et alia decided it was most plausible that an object of Mjölnir's mass would take time, in an atmosphere and having to pass through bulkheads, to arrive from wherever Thor had left it (presumably quarters had been set aside for each of them; at any rate, he didn't have it with him on the bridge, either).