I don't want to tread too far into the waters of gender politics or cause any arguments, but I shouldn't think it's too controversial to say it's an unfortunate trend that women in high office are often scrutinised much more heavily for their appearance compared to their male counterparts.
For instance, I think most Brits (whether for the blue team or the red team)* would agree Boris Johnson often looks a tad dishevelled, but it's not much of a talking point until he's outright stuck ziplining while waving about a pair of Union Jack flags. Theresa May on the other hand, despite being a female PM in the 21st century, was under far more scrutiny for how she looked and dressed during her premiership - see here, here, and here.
Gossip affects perception. If you hear something negative about someone, someone who you may even hold in a very positive light, the gossip nestles in your mind and places doubt within you. Couple this with the fact that the appearance of Harriet Jones is already under a harsher gaze by merit of her gender, and the person spreading the gossip is held in very high regard (the aide he whispers it to was present on the Sycorax ship when the Doctor whooped their leader, so has seen first-hand** what the Doctor is made of), and you've got a great way to undermine confidence in someone.
The Doctor, being brilliant, and with a penchant for humans and their behaviour as it is, knew that this sort of comment would be incredibly undermining. Yes, a bit of a dirty tactic to play on the psychology of humans and their early 21st century perception of female leaders, but she had just ordered the destruction of a fleeing spaceship and he was not particularly happy about it.
*Other teams available, enquire within
**Pun fully intended