The discussion below this question leads me to ask: What was the earliest SF work that used the idea of the Multiverse (parallel universes or alternate worlds)? I'm looking for fairly hard SF treatments, where the idea is presented as a real, physical possibility rather than magical or fantasy worlds, such as Narnia.
A robinsonade in the fourth dimension.
Jack Thorpe and Marjorie Matthews are walking along the shore in Cornwall when a seeming eccentric asks them directions. They humor him by showing the way, whereupon he produces a revolver and takes them captive. A scientist who has worked in dimensional research, he is brilliant, but unfortunately mad and irresponsible. He thinks of himself as a hubble-bubble, and so the characters call him.
The Hubble-Bubble reveals that he has obtained access, via the fourth dimension, to two other worlds which in modern terminology amount to parallel worlds. His technique involves electricity, vibrations, and a mental set. He now offers Jack and Marjorie the choice of death or entry to another world, which he claims is much like earth in fauna and flora, but without human or other intelligent life.
Jack and Marjorie have little choice, and in a short time find themselves in the world they later call Marjorie-land. Making the best of the situation, they work out a Crusoe-like primitive culture, building a house, cultivating certain plants, and domesticating animals. From time to time a few other humans are dropped in with them, a total of four batches in all. Most of them are congenial, but Michael Quelch, a lazy, vicious Cockney will eventually cause trouble.
On one occasion the Hubble-Bubble's apparatus seems to have "backfired," and Jack is temporarily returned to our world. But he makes terms of a sort with the mad scientist and goes back to Marjorie-land.
Time passes. Jack and Marjorie have two children, and the colonists thrive. Life seems reasonably secure and happy. But then Quelch causes trouble. Thinking that Jack is dead when he does not return on time from a journey of exploration, Quelch tries to seize control of the settlement, rape Marjorie, and murder the children. Fortunately, as in a novel by Edgar Rice Burroughs, Jack returns in the nick of time. After some complications Jack reluctantly sets out to hunt Quelch down and kill him, but Quelch is found dead of natural causes.
After a time it becomes apparent that the Hubble-Bubble is also dead and that the small human colony in Marjorie-land is permanently isolated.
The story is told by Thorpe, a generation or two later.
In the first part of the book, the treatment is flippant, but the story soon settles down to a rather dull development. Of historical interest as a very early parallel worlds story.
"Multi-" usually means "many"; not just "at least two." So if we interpret "Multiverse" to mean "lots of parallel universes are lurking out there, and are even accessible from here, at least occasionally," rather than just applying the word "multiverse" to a story in which it's possible to get from our world/universe/timeline to one other (which may be the only one that exists alongside our own), then I believe Murray Leinster may have been the first to go hog-wild with the "multiverse" idea in his story "Sidewise in Time," first published in Astounding Stories (June, 1934).
I don't believe he used the exact word "multiverse," but that was essentially what he was describing. In his take on it, the basic premise is that natural forces (which only one brilliant scientist had seen coming) start causing bits and pieces of our version of Planet Earth to suddenly switch places with the equivalent geographical areas on other versions of Planet Earth. Naturally, any humans standing on a particular bit of terrain at the time are taken along for the ride, whether they like it or not.
Alternate worlds which are glimpsed by viewpoint characters, or are at least mentioned in passing as having switched territory with ours (temporarily or permanently), include:
A world where Roman Legions are hanging out in the vicinity of what we call St. Louis.
A world where Vikings have at least one thriving colony on the Eastern Seaboard (and some of them start raiding the coast of Massachusetts).
A world where the Confederate States of America is still going strong as an independent nation in the 1930s (having won its big war about 70 years earlier, after a great victory at Gettysburg).
A world where the Chinese have long since colonized at least some portions of North America.
A world where Tsarist Russia controls what we call San Francisco (and has at least one outpost in Colorado, and who knows what else?).
Worlds where, as far as can be told, humans never evolved. Possibly several such worlds; I believe at least one timeline still had big dinosaurs running around.
And I'm sure I'm forgetting some others!
By the end of the story, the "sidewise in time" shifts have ended, and many of the stray bits and pieces of our world are back where they belong after temporary absences, but not everything has reverted back to "normal." In some cases, entire cities had been lost!
As far as I know, this was the first "fairly hard SF" story to take a careful look at the "multiverse" concept as I normally think of it today.
Ignoring fantasy elements, and looking purely at Alternate Earths:
The book by H.G. Wells Men Like Gods, which was published in 1923 features an alternate earth. Two people from our earth is transported to Utopia, which is 3000 years more advanced than our earth. There are mentions of differences in the world, such as Jesus dying on a wheel instead of a cross.