After the assassination attempt, it seemed obvious that Bran was pushed because he knew/saw something that he shouldn't have. Once it was clear that Cersei was cheating on Robert and bearing another man's children, it'd be a pretty good guess that Bran caught them in the act.
So from there, Eddard would only need to consider every member of King Robert's retinue who visited Winterfell. All of the adult noble men left Winterfell with the king and Eddard to go hunting during the hours that Bran was pushed... except for Jaime, who stayed behind to protect the queen. Jaime, who had previously shown himself capable of dishonorable violence. Jaime, his sister's stalwart defender. Jaime, who would have been in close proximity with the queen during the time of all three conceptions.
Couple that with how overwhelmingly all three Baratheon children had the Lannister look, and I don't think it's that hard of a conclusion to draw.
Also keep in mind, Eddard summons Cersei to the garden because he has discovered "the truth that Jon Arryn died for." Cersei is the one who starts talking about how Jaime would have killed Robert if he knew he hit her. It's that comment that leads Eddard to say, "Your brother... or your lover?" So it may be that Eddard summoned Cersei knowing that the kids weren't Robert's, but only suspecting that Jaime was the father. Cersei assumed Eddard knew everything, and basically telegraphed that they were lovers.
