The Universe Between, a 1965 novel by Alan E. Nourse, which was also the answer to this old question. It was based on the short story "High Threshold" in Astounding Science Fiction, March 1951 and the novelette "The Universe Between" in Astounding Science Fiction, September 1951, which are available at the Internet Archive here and here.
A scientist conducting experiments on low temperatures inadvertently trespasses on another universe.
I don't have a copy of the novel. The following excerpt is from the short story "High Threshold":
Ned McEvoy took the ball from his briefcase and laid it on the desk before the thin man with his pince-nez. "What does that look like to you, Dr. Bamford?"
The man examined it closely, and looked up smiling. "It looks like a tennis ball that someone has turned inside out," he replied.
McEvoy chuckled. "And how would you go about turning a tennis ball inside out, Dr. Bamford?"
"Can't say I know, offhand." He looked quizzically at McEvoy. "What can I do for you, doctor?"
"You've heard the old story of the goats that were carried across the Andes on muleback—and all died of fright?"
"Well, we have the same problem in my laboratory. Only we have men dying of fright."
Dr. Bamford's eyebrows went up. "Adjustment?" he ventured.
"We think so. About six months ago we ran into a peculiar snag in the work I was supervising. The Institute of Physics has been concerned for several years with problems involving extremely low temperatures—nearly absolute zero temperatures. The work we originally planned called theoretically for an approach within six decimals of complete cessation of molecular motion. That involves a temperature of one millionth of one degree Kelvin. And we reached it."
McEvoy fingered his collar nervously, and shifted his weight in the chair. "Matter of fact, we did even better. Our pumps began acting up as we approached a thousandth of a degree. What happened, we think, was a reversal of the Franklinson effect in the extremes of high temperature, where the temperature doubles quite suddenly with a tremendous molecular expansion. Our temperature took a sudden startling drop."
"I'm not quite sure I follow you," said Dr. Bamford. "From one degree Kelvin, just where does the temperature drop?"
McEvoy scowled. "A good question," he said. "I don't know, to be quite frank. Zero Kelvin is a relative and hypothetical point at which all molecular motion ceases. Below zero Kelvin, if such were possible, one could reasonably expect negative molecular motion. That may have been what we obtained; we don't know. But we certainly observed a change. The tungsten block we were treating simply evaporated. Vanished. The temperature recording device vanished. All we could see in the vault was a small glowing hole in the corner of the room where the block had been. Nothing in it, Dr. Bamford—nothing. And the hole, seen at a distance, appeared very suspiciously like a—hypercube."
The doctor was silent for a long moment. "You investigated?"
"We surely did. We're still trying. It looks as if we have a four-dimensional projection in our three-dimensional space—a corner, or an edge, of four-dimensional space. We've tried everything, and we're getting nowhere. So far we've lost five crackerjack technicians investigating. We're no better off than when we started."
And this one is from the novelette "The Universe Between":
He sat up, his eyes wide and frightened. "No wonder they're afraid! It's tearing their universe to shreds, warping hyperspace through hyperspace, turning the innertube inside out, inverting the circle! Your transmatter is twisting material objects through places where they simply can't go!" He jumped out of bed, his thin body trembling. "Can't you see it? All you get is a mild distortion of the objects that get through. But you're wrenching apart their whole universe! Of course they retaliate! What else can they do? They can bring force to bear, too, random distorting force on this side of the Threshold. They've known about the Threshold—I've been crossing through for years. They must have investigated it. And now they're clipping whole chunks of our universe right through into their hyperspace, trying to scare us, warn us off, stop us. They just happened to take San Francisco with them, and Boston was probably just a near miss! And those places are torn out by the roots, twisted up across the Threshold into a wild jungle of distorted geometry and raving lunatics."
The boy's face was white, his voice pleading. "Dr. McEvoy, you've got to turn that thing off. Before they split the earth in two."