The book Phantastes from 1858 seems to contain an early example, judging from this section of the wikipedia plot description:
Anodos escapes this place and finds himself in a stormy sea. When a boat arrives, he boards it. It takes him to an "island" with a cottage with four doors which is inhabited by an ancient lady with young eyes. Anodos enters each door in turn, each containing a different world. In the first he becomes a child again, remembering the death of his brother. He comes back to the cottage crying. In the next door he finds the marble lady and Sir Percivale, alive, well, and in love. They are talking about him, and Anodos (previously unnoticed) makes a last outburst of his love for the marble lady. They leave, as does Anodos. The next door recounts the death of a loved one of Anodos, and he finds his family mausoleum. His ancestors help him back to the cottage. Finally, Anodos travels through the last door ("the door of the timeless") but is saved by the ancient lady without remembering anything. The ancient lady says that because she saved him, he must leave (the "island" in fact has an isthmus).
The article also notes that C.S. Lewis recalled reading the book at age 16, and said that "That night my imagination was, in a certain sense, baptized", so it's possible it was an influence on his stories featuring magical portals.