I suspect that this is not the correct story because it isn't told in the first person, however the plot is basically similar and I wonder if it's related. If nothing else it might jog a memory somewhere.
Anyhow, What the EPA Don't Know Won't Hurt Them is a short story by Suzette Haden Elgin. It's a prequel to her Ozark trilogy, which is about twelve Ozark families who managed to travel to another planet and colonise it. In the short story a group of the Ozarks have been stranded on Earth when their spaceship broke down, and they're hunting for the parts to repair it.
They find the final part purely by accident when a child playing in a junkyard finds it. They rebuild their spaceship, and the next day they're gone.
Possible or not, there it was. "Twelve Arkansas families disappear from the face of the Earth overnight!" the newspapers screamed, using the biggest type they had available. "FBI estimates a thousand gone without a trace! Authorities baffled!" "Administration suspects terrorists!"
They were gone, and much of their belongings with them. All their houses and outbuildings were swept and tidy and still. On every kitchen table lay a neat stack of envelopes with bills inside, and checks or cash in each one to cover the obligation. Even the junk piled in the yards and ditches and ravines was tidy; the vegetation around it seemed to have all been burned away by the kind of fire that burns so hot it leaves not even ashes behind, though not a single fire had been reported. The junk itself looking burnished and shiny and sparkling, with no sign of the rust and filth that had been there the day before. But nobody had seen the Ozarkers leaving the hills. Nobody had seen them drive away, or get on a bus, or board a plane. Nobody'd sold them gas; nobody'd sold them tickets. Not one of them had given notices at the places where they worked, or offered any other warning. They were just GONE. As if they'd never been there at all.