During The Time of the Doctor it is revealed, that ...
the Silents were genetically engineered as "Confessors", so people could confess them anything and immediately forget about doing it, just because of their looks.
While at first I had an "ah!" moment, thinking about it really confuses me and it no longer makes any sense.
The basic idea behind confessing would be to tell someone about your bad deeds to get absolution. For this two work there's one central condition that has to be true for it to work: You have to know (or at least believe) that you've shared your bad deed with someone else and that you'll be forgiven despite doing it.
So how does this fit into the whole promise of the Silents? Let's say I confess something, then look away, and I immediately forget about confessing anything. That doesn't make any sense, because I'd want to confess again. You can compare this to the doctor (the medical/army one) nursing the Silent inside the black cube in Area 51 (The Day of the Moon). He helps the alien, but immedaitely forgets about doing it.
The opposite would clearly work: Someone you can tell your deepest secrets who'll then forget about it. They'd be the perfect keeper always respecting the confessional secret. But the other way around?