"The Man Outside", a short story by Evelyn E. Smith, is available at Project Gutenberg.

Some of Martin's descendants have come back through time to protect him:

Sometimes he wondered who Ninian really was. Obviously that story about her coming from the future was just a gag. Besides, if she really was his great-great-grand-daughter, as she said, why would she tell him to call her "Aunt Ninian"? Maybe he was only eleven, but he'd been around and he knew just what the score was. At first he'd thought maybe she was some new kind of social worker, but she acted a little too crazy for that.

He loved to bait her, as he had loved to bait his mother. It was safer with Ninian, though, because when he pushed her too far, she would cry instead of mopping up the floor with him.

"But I can't understand," he would say, keeping his face straight. "Why do you have to come from the future to protect me against your cousin Conrad?"

"Because he's coming to kill you."

"Why should he kill me? I ain't done him nothing."

Ninian sighed. "He's dissatisfied with the current social order and killing you is part of an elaborate plan he's formulated to change it. You wouldn't understand."

Martin's descendants are not very bright:

"I'm trying to talk to you as if you were an adult," Raymond said, "but if you will persist in these childish interruptions—"

"I'm sorry," Martin said.

But he wasn't, for by now he had little respect left for any of his descendants. They were all exceedingly handsome and cultivated young people, with superior educations, smooth ways of speaking and considerable self-confidence, but they just weren't very bright. And he had discovered that Raymond was perhaps the most intelligent of the lot. Somewhere in that relatively short span of time, his line or—more frightening—his race had lost something vital.

Unaware of the near-contempt in which his young ancestor held him, Raymond went on blandly: "Anyhow, Conrad took it upon himself to feel particularly guilty, because, he decided, if it hadn't been for the fact that our great-grandfather discovered the super-drive, we might never have reached the stars. Which is ridiculous—his feeling guilty, I mean. Perhaps a great-grandfather is responsible for his great-grandchildren, but a great-grandchild can hardly be held accountable for his great-grandfather."

Eventually they go to sea:

"'Course you can't go on the Grand Tour. Your son hasn't invented space travel yet. But we can go see this world. What's left of it. Tourists always like ruins best, anyway."

So he drew on the family's vast future resources and bought a yacht, which Martin christened The Interregnum. They traveled about from sea to ocean and from ocean to sea, touching at various ports and making trips inland. Martin saw the civilized world—mostly in fragments; the nearly intact semi-civilized world and the uncivilized world, much the same as it had been for centuries. It was like visiting an enormous museum; he couldn't seem to identify with his own time any more.

The other cousins appeared to find the yacht a congenial head-quarters, largely because they could spend so much time far away from the contemporary inhabitants of the planet and relax and be themselves. So they never moved back to land. Martin spent the rest of his life on The Interregnum. He felt curiously safer from Conrad there, although there was no valid reason why an ocean should stop a traveler through time.

As Martin is on his deathbed:

There was only one face which Martin had never seen before. It wasn't a strange face, however, because Martin had seen one very like it in the looking glass when he was a young man.

"You must be Conrad," Martin called across the cabin in a voice that was still clear. "I've been looking forward to meeting you for some time."

The other cousins whirled to face the newcomer.

"You're too late, Con," Raymond gloated for the whole generation. "He's lived out his life."

"But he hasn't lived out his life," Conrad contradicted. "He's lived out the life you created for him. And for yourselves, too."

For the first time, Martin saw compassion in the eyes of one of his lineage and found it vaguely disturbing. It didn't seem to belong there.

"Don't you realize even yet," Conrad went on, "that as soon as he goes, you'll go, too—present, past, future, wherever you are, you'll go up in the air like puffs of smoke?"

"What do you mean?" Ninian quavered, her soft, pretty face alarmed.

Martin answered Conrad's rueful smile, but left the explanations up to him. It was his show, after all.

"Because you will never have existed," Conrad said. "You have no right to existence; it was you yourselves who watched him all the time, so he didn't have a chance to lead a normal life, get married, have children...."

[. . . .]

Well, it didn't matter; whatever happened, no one could hold him to blame. He held no stake in the future that was to come. It was other men's future—other men's problem. He died very peacefully then, and, since he was the only one left on the ship, there was nobody to bury him.

The unmanned yacht drifted about the seas for years and gave rise to many legends, none of them as unbelievable as the truth.