I read a sci fi short story sometime between 1974-1980 but it may have been written long before that. I believe it was in a paper back anthology of sci fi stories, but I'm not certain.
"Wrong-Way Street", a short story by Larry Niven, first published in Galaxy Magazine, April 1965, which is available at the Internet Archive (click here for download options). Any of these covers look familiar?
The first flashback is to his childhood. He and his younger brother are sled riding. His brother crashes, ruptures his spleen and dies. He feels responsible and is wracked with guilt.
Young Tony had been riding a Flexy, a kind of bobsled on wheels, down the hill road above Venice Boulevard. At the bottom of the hill he had turned hard right onto Venice. The Flexy had flipped over on its back, and its blunt rubber handle had poked hard into Tony's stomach.
One of the first things the doctor did was to take Tony's blood pressure. It was low, which meant shock. It started to fall almost as soon as the blood pressure cuff was removed, but the doctor didn't know that until it was too late. Tony's spleen was ruptured.
Mike had loved his younger brother. He sat in his room most of the time, unable to get used to his loss, and not really trying. After four weeks of it his father neglected his own grief long enough to take Mike to a child psychologist.
Much later, as an adult, he is exploring a long-deserted alien base on the moon.
By the time Mike Capoferri graduated from high school he had become intensely interested in space travel. His first year at Cal Tech was the year Walnikov landed on Mars. Mike was determined to follow.
In a way he traveled farther than Walnikov. He never got to Mars, but he did make it to the Moon. And unlike Mars, the Moon once had intelligent visitors.
Mike was one of many. Thirty men and women had come to the alien base, determined to probe all its secrets. By this time Mike was thirty-one years old and held doctorates in physics, mathematics, and philosophy.
There are unfamiliar artifacts on the base, including a time-travel device/vehicle,
"Dust," Mike told himself. There was almost no moon-dust on the worn path between the ship and the base. Yet dust had spurted beneath his boots—and there was no Earth-built ship, and the station was locked.
"Eureka," he said softly. "They haven't found the base yet. I've traveled in time."
and a tool with an anti-matter tip. It carves neat holes and eliminates matter simply by moving it through matter.
I don't think antimatter is involved, just some unexplained advanced-technology magic.
A sculpting implement, used by the aliens as a means of recreation (the base was infested with the statues they had left behind), had become a disintegrator. Turning it on had been heartbreakingly difficult. Mike had solved that problem in his second year, but had never been able to turn it off. The alien rec room had to be kept in vacuo, with a separate airlock, because air disappeared into the little ball of nothing at the end of the sculpting tool.
He accidentally drops the tool, which plummets unimpeded to the center of the moon.
What should he write? "The world is my ash tray," he decided, and slammed his toe into a ledge. He threw both hands out to break his fall, and changed his mind too late. Horrified, he watched the sculpting pencil vanish into the floor. It left a neat cylindrical hole.
He realizes that over time, it will consume and destroy the moon, which will be catastrophic for Earth.
He realizes that after he returns to the "present" and sees the damage to the Moon and Earth. (At the time the story was written, there was a theory that the Moon was responsible for keeping the Earth from growing a big thick atmosphere and turning into another Venus.)
At first glance it looked like a walnut shell, but not quite, for the shape was wrong and the convolutions were too deep. What it really resembled was a deflated beach ball which somebody had crushed between his hands.
The Moon had had a long time to push itself through a sphere an inch and a half in diameter. Probably it had not taken more than a few millennia. Afterward there had been nothing but this crumpled ball of waste, too light and rigid for gravity to compress it further. For three billion years the Earth had been moonless.
He uses the time-travel device to travel to the moon at an earlier time so that he can prevent himself from dropping the tool.
The door opened and he jumped toward the control board. Already he was planning. He had to go back some time before his first arrival. Then—remove the sculpting tool from the rec room, or somehow scramble the controls of the base airlock, or leave a message for "himself" on the outer door. Anything to restore the past.
He makes bigger problems for himself, forcing him to travel back further and further in time.
Nothing that complicated, just one trip back . . .
He travels so far back that he encounters the original alien inhabitants of the base.
To his right, the Earth was a vast, incredible crescent—and the plain was full of ships. They were of many different sizes, but they all had the same lump cylindrical shape. Tiny figures moved among them.
Stuart was right, he thought idiotically. You go the wrong way on a one-way street, you've got to have accidents. He turned and ran.
Unfortunately, the alien inhabitants capture him, and the story ends.
Behind him the lock swung open. Two ten-foot tripods turned on each other and gestured rapidly, like nests of striking snakes. Then one of them hopped after him and picked him up.