I'm trying to find a short story from the 60's(?) about a future society dominated by mutant(?) children who've killed off all the adults except, for sentimental reasons, one.
That's sort of like "Small World" (aka "The Underdweller"), a short story by William F. Nolan, available at Project Gutenberg; first published in Fantastic Universe, August 1957, available at the Internet Archive. It has been reprinted many times; does any of these covers ring a bell?
The children are not mutants. The adults were not killed off by the children, but by aliens. The children tolerated the man at first, but as they got older they turned against him.
He thought once more about the beginning, six years ago—about why he was still alive. The alien ships had struck Earth suddenly, without warning. Their attack had been thorough and deadly. In a matter of hours, the aliens had accomplished their clever mission—and the men and women of Earth were destroyed. A few survived, he was certain. He had never seen any of them, but he was convinced they existed. Los Angeles was not the world, after all, and since he had escaped, so must have others around the globe. He'd been working alone in the drains when the aliens struck, finishing a special job for the construction company on B Tunnel. He could still hear the weird sound of the mammoth ships and feel the intense heat of their passage.
Hunger forced him out, and overnight he had become a curiosity. The last man alive. For three years, he was not harmed. He worked with them, taught them many things, and tried to win their confidence. But, eventually, certain ones came to hate him, to be jealous of his relationship with the others. Luckily, he had been able to escape to the drains. That was three years ago, and now they had forgotten him.
[. . . .]
Lewis Stillman knew that he was going to die.
The rifle was empty at last; the final bullet had been fired. He had no place to run because they were all around him, in a slowly closing circle.
He looked at the ring of small cruel faces and thought, The aliens have done their job perfectly; the stopped Earth before she could reach the age of the rocket, before she could threaten planets beyond her own moon. What an immensely clever plan it had been! To destroy every human being on Earth above the age of six—and then to leave as quickly as they had come, knowing that Earth's back had been broken, that her survivors would revert to savagery as they grew into adulthood.
Lewis Stillman dropped the empty rifle at his feet and threw out his hands. "Listen," he pleaded, "I'm really one of you. You'll all be like me soon. Please, listen to me."
But the circle tightened relentlessly around Lewis Stillman. He was screaming when the children closed in.