How does the Animus know which memories to show?
An early cutscene from Assassin's Creed II suggests that the Animus operator can search through a subject's DNA based on some parameters. This cutscene in particular suggests that Lucy is searching for bits of Desmond's DNA that match up with certain portions of Subject 16's:
A side conversation with Shaun Hastings in Assassin's Creed III further suggests that the Animus has some sort of control for moving forwards and backwards through a Subject's DNA timeline:
Transcript of the relevant bits:
Shaun: Ah, Desmond, there you are! Can I ask a favour?
Shaun: When this is all over, I'd like to try turning the dial back on the Animus. Like, all the way back. To the time of the First Civilization.
Desmond: You think it would work?
Shaun: There was no real loss of fidelity when you visited Altaïr. Then again, that was about a thousand years ago and I'm looking at going back at least seventy thousand more.
Okay, how does the operator know when to go looking for memories?
The Lost Archive DLC for Assassin's Creed: Revelations reveals that Abstergo learned about Ezio through one of Subject 16's ancestors:
Italian Man 1: Have you heard of the Assissino (Assassin)?
Italian Man 2: Si. (Yes.) Ezio Auditore.
Vidic: Find out about this "Ezio Auditore". I want his records on my desk by the end of the week.
So that's one way they could find out about individuals: trial and error.
Finding time periods is a little trickier, but not insurmountable. A good theory is that the historical record gives them an idea of the rough timeframe, either through direct writings (Like Altaïr's codex) or extrapolating from historical events (The sudden rise of the Borgias, for instance, is a good indication that a Piece of Eden is floating around). Once they have a timeframe, they locate the ancestor who lived during that timeframe.