"Of Missing Persons" by Jack Finney. See this question and answer. First published in the March 1955 issue of Good Housekeeping. The ISFDB bibliography page lists some collections you might have read it in.
the main character begins to think it's all been a ripoff, there's no such thing as aliens or space travel or whatever, gets impatient and leaves the building...
The dark barn was silent now, except for our breathing. Time ticked away, and I felt an urge, presently, to speak to whoever was next to me. But I didn't quite know what to say, and I began to feel embarrassed, a little foolish, and very aware that I was simply sitting in an old and deserted barn. The seconds passed, and I moved my feet restlessly; presently I realized that I was getting cold and chilled. Then suddenly I knew—and my face flushed in violent anger and a terrible shame. We'd been tricked! bilked out of our money by our pathetic will to believe an absurd and fantastic fable and left, now, to sit there as long as we pleased, until we came to our senses finally, like countless others before us, and made our way home as best we could. It was suddenly impossible to understand or even remember how I could have been so gullible, and I was on my feet, stumbling through the dark across the uneven floor, with some notion of getting to a phone and the police. The big barn door was heavier than I'd thought, but I slid it back, took a running step through it, then turned to shout back to the others to come along.
when he comes back they've all gone... ?? he has missed his opportunity.
It took four or five seconds, no longer, fumbling at that door, to heave it open again. But it was four or five seconds too long. The barn was empty, dark. There was nothing inside but a worn pine bench—and, in the flicker of the lighted match in my hand, tiny drifts of what looked like damp confetti on the floor. As my mind had known even as my hands scratched at the outside of that door, there was no one inside now; and I knew where they were—knew they were walking, laughing aloud in a sudden wonderful and eager ecstasy, down into that forest-green valley, toward home.
