In A Good Man Goes To War, they wonder about Amy's baby:
DOCTOR: It doesn't make sense. You can't just cook yourself a Time Lord.
VASTRA: Of course not. But you gave them one hell of a start, and they've been working very hard ever since.
DORIUM: And yet they gave in so easily. Does this not that bother anyone else?
DOCTOR: Amy. She worried the baby would have a time head. She said that
VASTRA: Only you would ignore the instincts of a mother.
DORIUM: Or the instincts of a coward. This is too easy. There's something wrong.
DOCTOR: Why even do it? Even if you could get your hands on a brand new Time Lord, what for?
VASTRA: A weapon?
DOCTOR: Why would a Time Lord be a weapon?
VASTRA: Well, they've seen you.
Then he sits having a spooked look on his face and says
which made me think why did the Doctor look scared after Vastra said "they've seen you"? Was it the dangerous things things he did in his past incarnations?