So, according to Tolkien, all his Middle-earth novels take place in the real world, about 6,000 years before they were released.
So my question is, did he ever state how he learned of the history of Middle-earth? I vaguely recall that the Hobbits told him but I don't know if that is true or not.
(I know that in real life, he just wrote the books himself, but I'm looking for in-universe answers.)
The full origin is revealed in the prologue to Fellowship of the Ring, where Tolkien discusses the Red Book of Westmarch. The Red Book is actually a collection of other works, containing all of the material for
Tolkien discovered a copy that had been made in Minas Tirith early in the Fourth Age (emphasis mine):
This account of the end of the Third Age is drawn mainly from the Red Book of Westmarch. That most important source for the history of the War of the Ring was so called because it was long preserved at Undertowers, the home of the Fairbairns, Wardens of the Westmarch. It was in origin Bilbo's private diary, which he took with him to Rivendell. Frodo brought it back to the Shire, together with many loose leaves of notes, and during S.R. 1420-1 he nearly filled its pages with his account of the War. But annexed to it and preserved with it, probably in a single red case, were the three large volumes, bound in red leather, that Bilbo gave to him as a parting gift. To these four volumes there was added in Westmarch a fifth containing commentaries, genealogies, and various other matter concerning the hobbit members of the Fellowship.
The original Red Book has not been preserved, but many copies were made, especially of the first volume, for the use of the descendants of the children of Master Samwise. The most important copy, however, has a different history. It was kept at Great Smials, but it was written in Gondor, probably at the request of the great-grandson of Peregrin, and completed in S.R. 1592 (F.A. 172). Its southern scribe appended this note: Findegil, King's Writer, finished this work in IV 172. It is an exact copy in all details of the Thain's Book in Minas Tirith. That book was a copy, made at the request of King Elessar, of the Red Book of the Periannath, and was brought to him by the Thain Peregrin when he retired to Gondor in IV 64.
The Thain's Book was thus the first copy made of the Red Book and contained much that was later omitted or lost. In Minas Tirith it received much annotation, and many corrections, especially of names, words, and quotations in the Elvish languages; and there was added to it an abbreviated version of those parts of The Tale of Aragorn and Arwen which lie outside the account of the War. The full tale is stated to have been written by Barahir, grandson of the Steward Faramir, some time after the passing of the King. But the chief importance of Findegil's copy is that it alone contains the whole of Bilbo's 'Translations from the Elvish'. These three volumes were found to be a work of great skill and learning in which, between 1403 and 1418, he had used all the sources available to him in Rivendell, both living and written. But since they were little used by Frodo, being almost entirely concerned with the Elder Days, no more is said of them here.
Fellowship of the Ring Prologue 5: Note on the Shire Records
This is further revealed in a message printed in Elvish characters on the inside cover of The Lord of the Rings books; as I discussed in another answer, that script can be translated as:
The Lord of the Rings translated from the Red Book of Westmarch by John Ronald Reuel Tolkien. Herein is set forth the history of the War of the Ring and the Return of the King as seen by the Hobbits.
There's an abbreviated version of this in Tolkien's Foreword to the first edition of the book; while he changed the foreword for later editions (because he felt it was confusing; I guess some people thought he was being serious?), a part of the original foreword is printed in The Peoples of Middle-earth, and the whole thing is included in Hammond & Scull's Reader's Companion:
This tale is drawn in large part from the memoirs of Bilbo and Frodo Baggins, preserved for the most part in the Great Red Book of Samwise. It has been written during many years for those who were interested in the account of the great Adventure of Bilbo, and especially for my friends the Inklings (in whose veins, I suspect, a good deal of hobbit blood still runs), and for my sons and daughter.
History of Middle-earth XII The Peoples of Middle-earth Chapter 2: "The Appendix on Languages"
What I'm not sure about is how Tolkien got his hands on the Red Book. Presumably he found some old jars in a cave somewhere in the English countryside.
Interestingly, Tolkien initially had a different origin in mind for The Silmarillion. For much of his life he thought of that work as having been compiled by Ælfwine, a 10th-century Englishman who accidentally got to Tol Eressëa and heard the story directly from the Eldar. Ælfwine's role is discussed extensively throughout the History of Middle-earth series, but Christopher Tolkien edited him out of the published Silmarillion.
In addition to the Red Book, which contributed the bulk of the narrative, Tolkien reveals in the foreword to the first edition (reprinted in The Peoples of Middle-earth) that he also drew from some surviving records of Gondor:
I have supplemented the account of the Red Book, in places, with information derived from the surviving records of Gondor, notably the Book of the Kings.
History of Middle-earth XII The Peoples of Middle-earth Chapter 2: "The Appendix on Languages"
These records were presumably the source of much of Appendix A, and possibly Appendix B.
In short, for the books about the Third Age, the front matter for LOTR offers the conceit that the Hobbits wrote the history down (in the "Red Book of Westmarch") and that Tolkien served as the translator. Peter Jackson's movies use the Red Book as a framing device; we see both Bilbo and Frodo writing on it.
This account of the end of the Third Age is drawn mainly from the Red Book of Westmarch. That most important source for the history of the War of the Ring was so called because it was long preserved at Undertowers, the home of the Fairbairns, Wardens of the Westmarch. Blockquote
So within the fictional universe Lord of the Rings is MOSTLY based on THE MOST IMPORTANT SOURCE The Red Book of Westmarch but a minority of it is based on one or more sources of lesser importance. Since Tolkien mentions that modern hobbits say that their ancestors used to be taller, perhaps the other sources are oral history stories among the hobbits and later hobbit writings based on The Red Book of Westmarch. in cases where those sources disagreed, The Red Book of Westmarch, being the oldest source, and a first hand account written by two of the five leading hobbits, would obviously be the preferred source.
We can wonder if Tolkien had other sources for The Silmarillion, for example. Comparing the various versions of the Silmarillion stories that Tolkien wrote, he may have had access to several different sources for them, and considered the latest written versions to be based on the most reliable source he found.
I presume either Ælfwine or Alboin wrote it down and Tolkien found it in a university library or something. Either that or like characters in Lost Road he had dreams about it. He would be contemporary to some of the dreamers so he could get information from them.
