Amy and Rory aren't living in a time-energy farm. It was destroyed when they jumped off Winter Quay. There's also a scene that was shot where a man shows up at Brian's door (about a week after The Power of 3, in his time) and explains to them what happened. He says that he is the adopted son of Amy and Rory, after they got sent to the past. So Brian doesn't spend his whole life wondering what happened, and they really did live out a good life together.
I wonder if Amy and Rory ever accidentally encountered him in the present. They wouldn't have known who he was, having not been sent back yet. And surely they raised their kid on the stories of their adventures. So he would know better than to interact with their timelines. I also wonder if he ever met River? They are siblings!
I agree though that the doctor should be able to visit them. He can't fly the Tardis into 1938 NYC anymore, but he should be able to land upstate in 1945 and take a train in. Or meet them somewhere outside the city at a later date. Even if somehow they couldn't travel with him anymore, he could still see them, right?
And how does River go about telling Amy to write the afterword? She must go into the past and meet her. Or at least send her a letter. And if she can send a letter to them in their time, then they could also hop a bus and see each other.