I remember reading at least one story (there's probably more than one use of this premise around) in which reality was literally shaped by human belief. So when people believed the Earth was flat, it was. When they believed in the ether, it existed. So when enough people believed the Earth was a sphere, it became one. Does anybody know the title?
I don't really imagine this is the story you seek, but Heinlein's Waldo contains a passage I've always liked, which beautifully encapsulates the worldview you describe:
Suppose Chaos were king and the order we thought we detected in the world about us a mere phantasm of the imagination; where would that lead us? In that case, Waldo decided, it was entirely possible that a ten-pound weight did fall ten times as fast as a one-pound weight until the day the audacious Galileo decided in his mind that it was not so. Perhaps the whole science of ballistics derived from the convictions of a few firm-minded individuals who had sold the notion to the world. Perhaps the very stars were held firm in their courses by the unvarying faith of the astronomers. Orderly Cosmos, created out of Chaos -- by Mind!
The world was flat before geographers decided to think of it otherwise. The world was flat, and the Sun, tub size, rose in the east and set in the west. The stars were little lights, studding a pellucid dome which barely cleared the tallest mountains. Storms were the wrath of gods and had nothing to do with the calculus of air masses. A Mind-created animism dominated the world then.
More recently it had been different. A prevalent convention of materialistic and invariable causation had ruled the world; on it was based the whole involved technology of a machine-powered civilization. The machines worked, the way they were designed to work, because everybody believed in them.
Until a few pilots, somewhat debilitated by overmuch exposure to radiation, had lost their confidence and infected their machines with uncertainty -- and thereby let magic loose in the world.
I remember reading at least one story (there's probably more than one use of this premise around) in which reality was literally shaped by human belief.
A classic science fiction story on that theme is the 1950 novelette "The New Reality" by Charles L. Harness (which may also be the answer to this old question), first published in Thrilling Wonder Stories, December 1950, available at the Internet Archive. A character in the story explains the premise:
The ontologist continued rapidly. "All of you doubt my sanity. A week ago I would have, too. But since then I've done a great deal of research in the history of science. And I repeat, the universe is the work of man. I believe that man began his existence in some incredibly simple world— the original and true noumenon of our present universe. And that over the centuries man expanded his little world into its present vastness and incomprehensible intricacy solely by dint of imagination.
So when people believed the Earth was flat, it was. [. . .] So when enough people believed the Earth was a sphere, it became one.
As I got deeper into the manuscript, my mouth grew dry, and my heart began to pound. This, I knew, was a contribution the like of which my family has not seen since Copernicus, Roger Bacon, or perhaps even Aristotle. It seemed incredible that this silent little man, who had never been outside of Koenigsberg, should hold the key to the universe—the Critique of Pure Reason, he calls it. And I doubt that even he realizes the ultimate portent of his teaching, for he says we cannot know the real shape or nature of anything, that is, the Thing-in-Itself, the Ding-an-Sich, or noumenon. He holds that this is the ultimate unknowable, reserved to the gods. He doesn't suspect that, century by century, mankind is nearing this final realization of the final things. Even this brilliant man would probably say that the earth was round in 600 B.C., even as it is today. But I know it was flat then—as truly flat as it is truly round today. What has changed? Not the Thing-in-Itself we call the earth. No, it is the mind of man that has changed. But in his preposterous blindness, he mistakes what is really his own mental quickening for a broadened application of science and more precise methods of investigation—
When they believed in the ether, it existed.
Check history of gravity—acceleration. Believe Aristotle unable detect acceleration. Galileo used same instruments, including same crude water clock, and found it. Why? . . . Any reported transits of Vulcan since 1914, when Einstein explained eccentricity of Mercury orbit by relativity instead of by hypothetical sunward planet? . . . How could Oliver Lodge detect an ether-drift and Michelson not? Conceivable that Lorentz contraction not a physical fact before Michelson experiment? . . . How many chemical elements were predicted before discovered?
A more recent story with a similar premise is the 1989 novella "Matter's End" by Gregory Benford. Both stories end with the-world-as-we-know-it destroyed as the result of a physics experiment: the disintegration of a photon in Harness's story, observation of proton decay in Benford's. However, the idea that the earth used to be flat, or that the luminiferous ether used to exist, is not mentioned explicitly in Benford's story, although the moon does assume a cubical form near the end:
Smooth glistening forms began to emerge from the rough, coarse earth. Above the riotous, heaving land the moon was now a brassy cube. Across its face played enormous black cracks like mad lightning.
This reminds me a lot of "The Vanished" by John Peel.
The story starts with a handful teenagers waking up to find the world empty, the people vanished. They eventually discover find out they are part of an experiment, testing that same theory -
they were dropped on an uninhabitable alien world, and since they weren't allowed to know about the experiment they believed a copy of their world into existence... cheap and easy terraforming.
The experiment proved the theory, in that all the kids survived the poisonous alien atmosphere, but ultimately failed because the world (cities, tech and all) was too complex, the kids were too few, and they discovered the experiment - which had depended on the unquestioning belief in their surroundings that knowledge of the experiment undermined. In the end, a couple of the kids manage to deliberately believe, and so create, a simpler but survivable garden world.
The theory (of reality conforming to belief) comes up during the story as a plot point - with some people [called cornerstones] being better at influencing the collective reality than others. The world was maintained through this combination of consensus and change so that, for example, the earth could transition from flat to round by individuals (including animals) being persuaded to believe or disbelieve those theories and 'discoveries'. It's spelled out pretty clearly in the text, but I don't have a copy on hand.
I'm not sure if this is the story you're looking for, since there's a handful of works with similar premises... but it seemed pretty similar and inspired me to find the story again, I hope it helps.