I think the story is By His Bootstraps by Robert Heinlein, but you have misremembered it as the story is far more complicated than you describe. It is a 20,000 word novella rather than a short story and explores some of the paradoxes inherent in time travel. It starts as you describe with a portal opening in the protagonists living room:
Bob Wilson did not see the circle grow.
Nor, for that matter, did he see the stranger who stepped out of the circle and stood staring at the back of Wilson’s neck—stared, and breathed heavily, as if laboring under strong and unusual emotion.
Wilson had no reason to suspect that anyone else was in his room; he had every reason to expect the contrary. He had locked himself in his room for the purpose of completing his thesis in one sustained drive. He had to—tomorrow was the last day for submission, yesterday the thesis had been no more than a title: “An Investigation Into Certain Mathematical Aspects of a Rigor of Metaphysics.”
And it has the scene where Mein Kampf is cut in half by the portal:
The Hall of the Gate was empty, to his great relief. What a break, he told himself thankfully. Just five minutes, that’s all I ask. Five uninterrupted minutes. He set the suitcases down near the Gate to be ready for a quick departure. As he did so he noticed that a large chunk was missing from a corner of one case. Half a book showed through the opening, sheared as neatly as with a printer’s trimmer. He identified it as “Mein Kampf.”
And the giant buildings on the other side of the portal:
The architecture of the place confused him. Not only was it strange to his experience—he had expected that—but the place, with minor exceptions, seemed totally unadapted to the uses of human beings. Great halls large enough to hold ten thousand people at once—had there been floors for them to stand on. For there frequently were no floors in the accepted meaning of a level or reasonably level platform. In following a passageway he came suddenly to one of the great mysterious openings in the structure and almost fell in before he realized that his path had terminated. He crawled gingerly forward and looked over the edge. The mouth of the passage debouched high up on a wall of the place; below him the wall was cut back so that there was not even a vertical surface for the eye to follow. Far below him, the wall curved back and met its mate of the opposite side—not decently, in a horizontal plane, but at an acute angle.
But it isn't a beast in a pit that nearly drives him mad. It's when he attempts to research the High Ones who built the palace:
When he pulled himself together he was halfway down the passageway leading away from the hall. He realized that he had been screaming. He still had an attack of the shakes.
Somewhat later he forced himself to return to the hall, and, with eyes averted, enter the control booth and return the spheres to zero. He backed out hastily and left the hall for his apartment. He did not touch the controls or enter the hall for more than two years.
It had not been fear of physical menace that had shaken his reason, nor the appearance of the creature—he could recall nothing of how it looked. It had been a feeling of sadness infinitely compounded which had flooded through him at the instant, a sense of tragedy, of grief insupportable and unescapable, of infinite weariness. He had been flicked with emotions many times too strong for his spiritual fiber and which he was no more fitted to experience than an oyster is to play a violin