"Passerby," a short story by Larry Niven. (As Joe Fromm suggested in a comment.) First published in Galaxy Magazine (September 1969). I first encountered it in the Niven collection All the Myriad Ways.
The plot is pretty much as described in the summary you quoted. But the story arguably has two narrators, depending upon how rigidly we define the term "narrator." The story as a whole is narrated by an unnamed man who spends a lot of time in a public park. His attention is caught by another man, a stranger to him, but obviously a "rammer." (A pilot who flies a starship equipped with a Bussard ramjet as its main power source.)
The rammer and the narrator both watch a little boy kneel down in a pathway in the park, and then stand up and move, with his hands cupped together, towards an oak tree. The rammer faints. His body starts to collapse and the narrator grabs him and manages to get the guy over to a park bench so he doesn't hit his head against the ground.
Then the bulk of the story is a conversation between them, with the rammer describing a very strange experience he had in space, so you could say that his dialogue qualifies as him narrating a story within the larger story narrated by the first guy. As I said, the plot is basically what was described above (although the golden giant appeared to be walking between the stars while standing upright, rather than making swimming motions). The rammer and his damaged ship were carefully dropped off in North Africa -- twelve light-years from where the rammer had been stranded in his ship -- and it seemed they had crossed that gap in mere moments. (Humans in this story had colonized other planets in other solar systems, but they did not have faster-than-light technology, and they didn't think anyone else did, either.)
The rammer and the narrator toss a few ideas back and forth regarding what that golden giant might have been. For instance, was it a mile-tall alien entity in approximately the shape of an adult male human (but with variations, such as three-fingered hands and no toes or visible ears), or was it a man-shaped starship that was built to look like a gigantic replica of its designer? At the end, the rammer explains why the sight of that boy's activities at the start of the story had triggered a panic reaction in him.
What the rammer had observed earlier, and the narrator hadn't, was that a caterpillar had been crawling along the gravel pathway, and various adults had come and gone without even noticing it. Then the boy came along, saw the caterpillar, and felt the urge to very carefully pick it up and move it off the path, over to an oak tree where he placed it on one branch, thinking it ought to be relatively safe up there. Then (implicitly) a very scary parallel between that episode and the rammer's experience suddenly occurred to the rammer, and so he collapsed.
I will quote the last paragraphs of the story. The narrator is speaking first.
"And you fainted."
"I should not have been so affected by what, after all, is no more
than a comparison. I would have cracked my skull had you not caught
me."
"A poor return for the golden one if you had."
The rammer did not smile. "Tell me . . . if an adult had seen the
caterpillar, instead of a boy--"
"Probably he'd have stepped on it."
"Yes, I thought so." The rammer put his tongue in his cheek, which
stretched incredibly. "He is nearly upside down. I hope he will not
fall off."
"It won't."
"Do you think he is safe there?"
"Sure. Don't worry about it."
The obvious implication is that it has just now occurred to the rammer that he may have been even luckier than he previously realized. What if the "golden giant" who rescued him was a particularly young and/or altruistic member of his species, and a more typical specimen would have used the rough equivalent of a fly swatter to deal with a useless little pest? What if that's the same attitude the alien adults would take towards human civilization in general -- if we ever managed to sufficiently annoy them so as to draw their attention for a few minutes?
