Why didn't River recognize him?
Because he's not supposed to exist. This is highlighted at least once in the episode itself:
[Alphonse pulls out a stack of pictures of the Doctor, from William Hartnell to John Hurt to Matt Smith]
Alphonse: None of these men are here. Are you sure it's one of these?
River: Yes. He only had these twelve faces.
[...]
Doctor: What if he has a face that you don't know about yet?
River: He has limits.
Doctor Who "The Husbands of River Song"
It's truly astonishing the kind of mental leaps your brain will go through trying to ignore something you know to be impossible; this is something the Doctor himself has commented on at least once, and has experienced on many occasions (albeit usually very briefly). Even if River's instincts are screaming out to her that this man is familiar, he can't possibly be the Doctor.
Why does she recognize him at that moment?
There was no specific prompt, but it seems to have been related both to her monologue about what it's like to love the Doctor, and the Doctor's visible reaction to what is simultaneously:
- Very touching (he had his moment of doubt about her earlier, but this monologue is about as romantic as you get when you're River and the Doctor)
- A bit insulting (since she outright says that it's not possible for him to love her, something he seems rather upset about)
So yeah, just the pieces sliding together. The capstone is probably the Doctor's response when she seems to have figured it out:
Doctor: Hello, sweetie.
Doctor Who "The Husbands of River Song"
At that point, who else could it be?