If you watch the beginning of the scene carefully, whatever the operative shoots Malcolm in the back with, it produces some sort of electrical discharge. I would presume that it was some sort of incapacitating type of round, rather than a penetrating type of round. This could be further reinforced by the moved nerve cluster, in that Malcom's nerves just don't react as expected.
Also, remember the line that the Operative gives before Serenity comes out of the cloud towing Reavers - "We should have done this as men...not with fire". He prefers the face to face combat, rather than the sterility of guns and lasers.
In the second instance, with the sword, if you watch the above video clip in a frame by frame type of fashion (from ~ :33 - :38), it shows that the sword goes through his (Malcom's) right side, fairly close to the edge of the body, and from a mid to upper angle.
If you look at the above image, on the right side, presuming that the angle of the sword is such that it stays below the rib cage, then about the only thing that would be penetrated would be the descending portion of the large intestine. While it is a wound that can produce nasty infections, it won't necessarily be incapacitating nor produce a ton of blood (And it is very possible that the shirt would clot to it, it isn't a large wound from top to bottom.)
The nerve cluster gets explained in the film. I will give Serenity credit in that Mal limps away with more than the usual movie hero damage.