Because at this point the Tams are only barely crew and Book is about to die
At this point in the show, Simon is moderately useful as a doctor and Mal sees River has a huge liability to the crew. Mal makes this clear in the opening sequence:
So, she's added cussing and hurling about
of things to her repertoire. She really
is a prodigy.
It's just a bad day.
No, a bad day is when someone's yellin'
spooks the cattle. Understand?
You ever see cattle stampede when they got
no place to run? It's kind of like a...
a meat grinder. And it'll lose us half
She hasn't gone anywhere near the cattle.
No, but in case you hadn't noticed, her voice
kinda carries. We're two miles above ground
and they can probably hear her down there.
Soon as we unload, she can holler until our
Although I would take it as a kindness if
(as a matter of reference, all quotes are from the same source)
Mal later basically warns Simon that he is willing to leave the Tams. Remember Mal is actually not known as a "nice guy" and loves to make subtle threats.
Closest Alliance is the Cruiser Magellan.
Hours out from here. And I promise you,
they ain't coming to a backwater like Jiang.
Still, I'm not sure it's such a wise suggestion.
Might not wanna mistake it for a suggestion.
Don't worry, we won't leave without ya.
Once Simon is kidnapped, Mal is left with a decision: a) Make the best bet on saving Book or b) risk Book's life and go try to rescue the Tams. As captain he calculates (reasonably) that he can find another doctor but that Shepard, quite like a companion, is a resource unlikely to ever fly with him ever again. He chooses Book.
And now they got themselves a doctor.
And we don't.
Mal moves to the cargo bay controls, begins shutting the bay door.
Wh-what are you doing? What about
Simon and River?
Forget them. We already lost two
people today. If I can help it,
we won't lose a third.
Wash, get us in the air.
The end dialogue might make it seem as though Mal had a different opinion all along:
Captain... why did you come back for us?
You're on my crew.
Yeah, but you don't even like me.
Why'd you come back?
You're on my crew. Why we still talking
(walks off, over his shoulder)
Chow's in ten. No need to dress.
But it is very important to remember that this comes after the conversation with both Jayne and Zoe ... two people who don't typically agree on the finer points:
That'd be a hell of a lot easier to do without
the two most wanted on board. Life would look
to be simpler us not carrying fugies.
He is right, you know.
In other words, the conversation at the end with Simon is confirmation that Mal has decided after the events of Safe that they are worth at least some risk. He might not entirely trust them, but he does accept them as crew. This episode is basically a transition for the Tams from being maybe crew to probably crew (sorry, Mal lives in a world of grey).
TLDR Until the events of Ariel, the Tams exist as somewhere between a commodity and a risk for Serenity and Mal knows that. Simon is clearly a commodity, but when you add River - who might ruin Mal's plans at any moment - and the risk of legal pursuit, they're pretty net neutral in terms of helping Serenity keep flying until Mal learns to trust them better.