It is very similar to "The Dandelion Girl" by Robert F. Young.
It was asked about here.
He was on vacation in a cabin; his wife could not join him that year. The much younger woman - over twenty years younger - who meets him on walks in the hills around the cabin evokes strong feelings in him. He does not find out for a while: it turns out that she is time traveling from the future.
Why, I'm forty-four, he thought wonderingly, and she's hardly more than twenty. What in heaven's name has come over me? "Are you enjoying the view?" he asked aloud.
There are further details in the story about her father who invented or perfected time travel, and how the technology, and her chance to travel back again, is at risk, and sharply limited.
However, while it is still possible, she can of course travel back to any point in her past.This provides for the twist ending, and his encounter with her at two different ages, over twenty years apart, in her life.
She is both the young girl in her twenties he meets who reminds him of his wife, and his wife in her forties, who had to keep her time travel a secret from him (and everyone). For over twenty years....
Until after he learns, in his own timeline, by the chance find back at his home, that "they" are the same person.