In A Good Man Goes To War, they wonder about Amy's baby:
DOCTOR: It doesn't make sense. You can't just cook yourself a Time Lord.
VASTRA: Of course not. But you gave them one hell of a start, and they've been working very hard ever since.
DORIUM: And yet they gave in so easily. Does this not that bother anyone else?
DOCTOR: Amy. She worried the baby would have a time head. She said that
VASTRA: Only you would ignore the instincts of a mother.
DORIUM: Or the instincts of a coward. This is too easy. There's something wrong.
DOCTOR: Why even do it? Even if you could get your hands on a brand new Time Lord, what for?
VASTRA: A weapon?
DOCTOR: Why would a Time Lord be a weapon?
VASTRA: Well, they've seen you.
Then he sits having a spooked look on his face and says
DOCTOR: Me?
which made me think why did the Doctor look scared after Vastra said "they've seen you"? Was it the dangerous things things he did in his past incarnations?