This is a short story that became an episode of The Twilight Zone. The man finds people waiting around a boxcar. He enters the box car, changes his mind, jumps off, then the door closes. He can’t get it open. Lights flash inside, then the door opens and the passengers are gone.
You're probably thinking of "Of Missing Persons" by Jack Finney.
As he sits and waits in the dark barn, Charley descends into a rage after he concludes he has been played for a fool. He storms out of the barn, but just as he crosses the threshold, he looks back and briefly glimpses, in a flash of light, the planet Verna through the back window of the barn before the barn door slams shut. By the time he gets the barn door back open, the people he left in the barn are gone, taken to Verna. Returning to the travel agency some time later, Charley is greeted by the proprietor, who hands him his money and says, "You left this on the counter last time you were here. I don't know why."
It's not a boxcar and it was not adapted into a Twilight Zone episode, although "A Stop at Willoughby"" involves trains and similar regret at missed opportunities.
FWIW, I remembered the plot of "Of Missing Persons" and did a search for
story-identification questions involving a travel agency, which led me to Short Story About a "Travel Agency" And A "Simple Life" Paradise.