This is definitely "Out of this World" (1964) by Alfred Bester. It was published in his 1964 collection The Dark Side of the Earth.
Howard Campbell introduces himself as someone who is faithful to his wife, but has a bit of a roving eye and will "fall in love" with random women he meets. In particular he tends to build fantasies about female wrong-number callers:
Wrong numbers are always the strongest temptation. The phone rings, I pick it up, a girl says, "May I talk to David, please?" There's no David in our house and I know it's a strange voice, but thrilling and tempting. In two seconds I've woven a fantasy of dating this stranger, meeting her, having an affair with her, breaking up my home, running off to Capri and living in glorious sin.
But he doesn't do anything about it until one day at work he gets a wrong-number call from a woman named Patsy - who ends up connected to him again when she tries the number a second time.
The next day Patsy calls again, still trying to reach "Prescot 9-3232" (Howard's office is "Plaza 6-5000"), and Howard asks her to lunch, which she declines. His secretary informs him that there are problems with the lines in their district, and the phone company is working on it.
Patsy calls again, and Howard convinces her to meet for lunch at Rockefeller Plaza:
"All right," she said. "My hour's one to two. Where shall we
meet?"
"Rockefeller Plaza. Third flagpole from the left."
But Patsy doesn't show up, and then she calls Howard and complains that he stood her up, and further that "Plaza 6-5000" doesn't exist, so he's lying to her and playing some kind of game. He convinces her to try meet him again the next day, near her work at "the old Tiffany building on Fifth," but again she doesn't show.
Howard tries to call her, but "Prescot 9-3232" doesn't exist, so he goes out to have a drink. Calling his office to check for messages, he ends up connected to Patsy again, and once again she claims she was waiting for him and he didn't show. Howard tries to figure out why she wasn't waiting, and figures out they live in different worlds:
"The old Tiffany building is at Fifty-seventh and Fifth."
"No, idiot! That's the new one."
"It's the old one. You know they had to move, back in 1945."
"Move?"
"Yes. They couldn't rebuild on account of the radiation."
"What radiation? What are you-"
"From the bomb crater."
A chill ran down my spine, and it wasn't from the damp and the cold. "Patsy," I said slowly. "This is serious, dear. I think maybe something more than telephone wires have been crossed. What's your phone exchange? Never mind the number. Just tell me your exchange."
"America 5."
I looked at the list of exchange names before me in the booth: ACademy 2, ADirondack 4, ALgonquin 4, ALgonquin 5, ATwater 9... There wasn't any AMerica 5.
"Here in Manhattan?'
"Of course, here in Manhattan. Where else?"
"The Bronx," I answered. "Or Brooklyn or Queens."
"Would I be living in occupation camps?"
I took a breath. "Patsy, dear, what's your last name? I think we'd better be honest about this because I think we're involved in something fantastic. I'm Howard Campbell."
She gasped.
"What's your last name, Patsy?"
"Shimabara," she said.
"You're Japanese?"
"Yes; You're Yank?"
"Yes. Were you born here, Patsy?"
"No. I came over in 1945 - with the occupation unit."
"I see. We lost the war - where you are."
"Of course. That's history. But, Howard, I'm here. I'm here in New York. It's 1954. It-"
"But the sun is shining and you dropped the A-bomb on us and licked us and you're occupying America." I began to laugh hysterically. "We're on different time tracks, Patsy. Your history isn't my history. We're in alternate worlds."
