From what I understand, The Lord of the Rings is written as the mythic past of our own world. We are constantly reminded that the race of man is becoming more and more prominent. At the end of The Return of the King the Elves leave Middle-earth to the humans. But what about the other mythical races, primarily the Hobbits and the Dwarves? Did they die out?
According to the prologue of the Lord of the Rings under the section Concerning Hobbits, the Hobbits are still with us today, but "now they avoid us with dismay and are becoming hard to find."
In Appendix A of the Lord of the Rings under section III, Durin's Folk, the following is mentioned as a possible reason for the lack of Dwarves:
It is because of the fewness of women them that the kind of Dwarves increases slowly, as in peril when they have no secure dwellings. For Dwarves only take one wife or husband each in their lives, and are jealous, as in all matters of their rights. The number of dwarf-men that marry is actually less than one-third. For not all the women take husbands: some desire none; some desire one that they cannot get, and so will have no other. As for the men, very many also do not desire marriage, being engrossed in their crafts.
There's no mention of the ultimate fate of the Dwarves, but it's easy to infer that Dwarves could not maintain a positive population growth rate.
First of all, to clarify a misconception:
"From what I understand LOTR is written as the mythic past of our own world."
No. LOTR setting is a totally different version of our world called "Arda", NOT "past".
Here's a supporting Tolkien quote I found on one forum:
Tolkien gave a radio interview for BBC 4 in 1971. The following is a question he was asked and his answer:
"'G: It seemed to me that Middle-earth was in a sense as you say this world we live in but at a different era. '
Tolkien: 'No ... at a different stage of imagination, yes.' "
So, dwarves and hobbits didn't "exist in the past" as per LOTR - they existed in different "mythology"
Also, as far as proposed reconciliation of hobbits and modern history, Encyclopedia of Arda has to say this:
Of Hobs and Boggarts
Throughout northern Europe, there exists a prevailing tradition of 'Little People'. They have an endless list of names: brownies, pixies, fays, leprechauns are just some of the more common. In some regions, these beings are far more than just myths or folklore: even today, they have an effect on people's everyday lives.
Take, for example, the Isle of Man in the middle of the Irish Sea: an island with a severe fairy infestation. In the southern parts of the island is the 'Fairy Bridge', a bridge that no Manxman would cross without greeting the Little People that live there. To most, of course, this is just superstition, but there are those who literally believe that they share their island with all manner of fairy creatures. Among these is a being known as a phynnodderee; shy of humans, friendly and happy-go-lucky, hairy-legged, fond of wine and beer and given to farm-work. Sound familiar?
The Manx aren't alone, of course: from Germany, where miners are helped by friendly burrowing 'kobolds', all the way to Iceland, whose Elves occupy a ghostly realm curiously similar to Tolkien's 'wraith-world', there are similar traditions.
What's more, even their names are familiar: we've already mentioned hob, but boggart, boggard, flibbertigibbet and even Hobberdy, Hobbidy and Hobberdy Dick (these last three are listed by Tolkien himself; The Letters of J.R.R. Tolkien, No 319, dated 1971).
Perhaps surprisingly, Tolkien denies that he was influenced by this in choosing the name 'hobbit', but he seems to have embraced the tradition by the time he wrote the Foreword to The Lord of the Rings. There, he says that hobbits are 'more numerous formerly than they are today', and that they 'avoid us with dismay and are becoming hard to find'. We can only realistically see this as an attempt to marry his fictional people with the 'hobbits' of folklore and tradition.
Although nothing is said in LotR, HoME 12 (The Peoples of Middle Earth) has this to say about the eventual fate of the Dwarves:
And the line of Dain and the wealth and renown of the kingship endured in Erebor until the world grew old, and the days of the Dwarves were ended.
A further revision brings the text to:
And the line of Dain prospered, and the wealth and renown of the kingship was renewed, until there arose again for the last time an heir of that House that bore the name of Durin, and he returned to Moria; and there was light again in deep places, and the ringing of hammers and the harping of harps, until the world grew old and the Dwarves failed and the days of Durin's race were ended.
Both of these passages are from original drafts of the "Durin's Folk" appendix, and while neither were taken up in the published text (and so their status as a canon or final decision on the matter must be considered dubious) they do imply that the Dwarves did eventually die out.
What was taken up in the published text was the concept of "Durin VII (and last)" who was a descendant of Dain Ironfoot and lived sometime in the Fourth Age, which seems to indicate that at least some of the concept of the Dwarves dying out was retained.
That however is not everything, for the Silmarillion also has this to say:
Aule the Maker, whom they call Mahal, cares for them, and gathers them to Mandos in halls set apart; and that he declared to their Fathers of old that Iluvatar will hallow them and give them a place among the Children in the End. Then their part shall be to serve Aule and to aid him in the remaking of Arda after the Last Battle.
So even assuming that the draft texts were never explicitly abandoned (but rather omitted for reasons of space, forgetfulness, or whatever) and assuming that the Dwarves did die out, then they will return at the End of Time and help rebuild the world.
One final passage, this time from Of The Rings of Power and the Third Age (published in the Silmarillion) may be mentioned:
Many things of beauty and wonder remained on earth in that time ... Dwarves still laboured in the hills and wrought with patient craft works of metal and stone that none now can rival.
Although not a conclusive statement, this can also be read as supporting the theory that the Dwarves died out, and when all the evidence is taken together it seems at least quite likely that this is what actually happened.
As I've understood it, with reference to the Dawn of Man / the Fourth Age, it seems that their supreme being (Eru) saw that Men would take over the world, so he willingly removed his mythical races from Middle Earth and delivered them to Valinor.
After the rebuilding and the environmental recovery of the Shire, the Hobbits flourished and had favor with the King of the reunified Kingdoms of Gondor and Arnor. But with all things that are in the world of man, evil caused the land and possessions of the Hobbits to become coveted and slowly they were pushed further west until they came under the protection of the Círdan in Mithlond and various Elven settlements beyond the Ered Luin (Blue Mountains). Some say Dwarves resettled there in the southern Ered Luin and also granted protection to the remaining Hobbits. But as the Elves finally left Middle-earth and the last Dwarves past into memory the remnant of the Hobbits were granted passage to a land beyond the seas where they flourished once again in their simple ways giving all who new them a warm heart, a seat near a warm hearth, a cup of tea and of course some fine leaf in a pipe to smoke.