tl;dr: He most likely "figured it out" once he pieced together what had happened to turn the Church into the Church of the Silence. Prior to that, their behavior would have been decidedly non-priest-like, thus making it difficult for him to make the connection.
First of all, note that The Doctor, even now, still can't possibly know what the priests look like, or remember anything that he learned about them while looking at their images. So, his information about their genetically engineered purpose must have come from somewhere else. The most likely answer is the Church records themselves, which he could have gotten access at any time from Tasha Lem. We don't see much chance for him to do so on Trenazlore, though its possible he did some digging while onboard the ship but off-screen.
But, even if he did know about them before Trenazlore, it's likely he wouldn't have identified them as the same creatures he fought on Earth. The ones that were working with Madame Kovarian were definitely not behaving like confessor priests. They were messing with Earth's history and trying to kill The Doctor and his friends. Since he doesn't actually know what the priests look like, when he saw one in Earth he wouldn't recognize it. Its behavior doesn't sound anything like what you'd expect from a priest, and at that point in his timeline, he didn't know what "The Church of the Silence" actually was. Thus, there wasn't really much to connect the two creatures together.
Based on his actions in Time of the Doctor, it seems like he didn't make the final connection until he was on Trenazlore. If nothing else, once he heard Tasha Lem rename the Church into "The Church of the Silence", he would have figured out the rest. He still wouldn't recognize such a priest if he saw it, but when Clara asked about those weird creatures, walking around the Papal Mainframe, that she keeps forgetting about as soon as she looked away, The Doctor would recognize those as the signs of a priest being nearby.
The major flaw in this logic, of course, is that these creatures are the only one we've seen thus far with the memory-redacting effect on others. That seems unique enough to trigger some red flags in The Doctor's mind. However, note that The Doctor does say that people tend to forget everything about them after not seeing them for a while -- Canton Delaware forgets they even exist after a short time. Thus, even if The Doctor did make the mental connection to the ones on Earth, he would likely forget that connection very quickly. It was only when he made the connection through some other avenue, as above, that it finally stuck.
