Are you remembering "The Impacted Man" by Robert Sheckley?
"The Impacted Man" was first published in Astounding Science Fiction Vol. L, No. 4 (December 1952) and included in the Sheckley's first short story collection, Untouched By Human Hands in 1954.
RE: a man who is able to travel back/forth in time by physically going down/up
The protagonist, Jack Masrin, is the titular impacted man, who travels through time when changing elevation in his brownstone. He first encounters this descending the stairs...
He reached the eighth step, and— He was standing on a grassy plain.
The transition was as sudden as that. He gasped and blinked. The suitcase
was still in his hand. But where was the brownstone? Where was
Kay? Where, for that matter, was New York? In the distance was a small
blue mountain. There was a clump of trees nearby. In front of the
clump was a dozen or so men. Masrin was in a dreamlike state of shock.
He observed, almost idly, that the men were short, swarthy, thickly
muscled. They wore loin cloths, and carried beautifully carved and
polished clubs.
RE: Part of the story is composed of the humorously bureaucratic letters exchanged between the entity that created this universe and the entity who commissioned the work.
The prose is interspersed with epistolary transmissions between Contractor Carienomen (whose team designed and built the metagalaxy) and Controller Miglese (who commissioned it).
RE: the ending
“Good boy!” Harf said. “Hand it here!” Masrin handed him the club. He
went over to Kay and put his arm around her. It was a paradox now, as
certainly as if he had killed his great- great-grandfather before he
was born. “That’s a lovely thing,” Harf said, admiring the club under
the light. “Consider your rent paid for the rest of the month—.” The
club disappeared from his hand. Harf disappeared.
You can read "The Impacted Man" on the Internet Archive site.
