"The Communicator" in particular
Early dialogue indicates that the sensors on the ship simply aren't accurate enough to lock onto Malcolm's communicator; they only get a definite position when they use hand-scanners while on the planet itself:
Hoshi: I've isolated the signal to within three city blocks. That's the best I can do, sir.
Star Trek Enterprise Season 2 Episode 8: "The Communicator"
In general
Generally speaking, it's mistrust of the technology. Malcolm and Travis discuss this in the first episode:
Travis: I heard this platform's been approved for bio-transport.
Malcolm: I presume you mean fruits and vegetables.
Travis: I mean Armoury Officers and Helmsmen.
Malcolm: I don't think I'm quite ready to have my molecules compressed into a data stream.
Travis: They claim it's safe.
Malcolm: Do they indeed. Well, I certainly hope the Captain doesn't plan on making us use it.
Travis: Don't worry, from what I'm told, he wouldn't even put his dog through this thing.
Star Trek Enterprise Season 1 Episode 1: "Broken Bow"
Later in the episode, Archer reveals just how little faith he has in the machine:
Malcolm: We could always try the transporting device.
Archer: We've risked too much to bring him back inside out.
Star Trek Enterprise Season 1 Episode 1: "Broken Bow"
And, in a later episode, Phlox discusses human apprehensions towards new technology after Hoshi reports feelings of unease following her first transport:
Phlox: Transporter technology is very new. I'm sure humans were equally frightened when the automobile was introduced, or the airplane. New forms of transport take a while to get used to. I'm not at all surprised at your reaction. You wouldn't catch me using that apparatus. But I can promise you one thing. You're in perfect health.
Star Trek Enterprise Season 2 Episode 10: "Vanishing Point"
It's worth noting that Phlox is absolutely correct in his assessment: when steam-powered trains were first introduced, there was a widespread belief that travelling at that speed would cause womens' uteruses to fly out of their bodies; humanity's irrational fear of the unknown is nothing new.