Why is there darkness/evil in Middle-earth and why couldn't the Valar fix this?

Question

As I've been re-reading The Lord of the Rings, The Silmarillion and now The Hobbit, I've come to a conclusion:

Basically only Hobbiton (not even all of the Shire) is a nice place. Well, also Lothlórien, obviously.

But the rest is mostly a wasteland or even worse. Just the fact that, that gigantic spider exists at all, no matter how far away and no matter how restricted to a certain area in Mordor, gives me the creeps more than anything else. She, and her disgusting babies in The Hobbit, is the real evil as far as I'm concerned. Even Sauron and Melkor have nothing on those disgusting spiders.

There are numerous very frightening entities and locations everywhere, with only small spots that I'd call habitable.

It never seems to be explained why the Valar don't fix this, or can't fix it. Much like in the real world, I fundamentally don't understand why Eru never seems to correct his mistake in creating that Melkor entity which originally started all of the evil stuff (including the spiders).

Why can't Middle-earth be just like Valinor/Aman? Even the cozy and safe Hobbiton seems to be trash compared to that place! Why must there necessarily be all of this darkness, evilness, misery, death, torture, fear, etc.?

Answer 1

The Marring of Arda was intentional

The idea of Arda Marred (as it's commonly referred to) is a direct reflection of the downfall of Man in the bible and the imperfections of the world. Similar to the bible, the history of Arda Marred ends with an apocalyptic event — Dagor Dagorath — after which the People's of Ëa begin living in what is known as Arda Unmarred (or Arda Healed).

Origins of the evils

It is important to know how the world (or Ëa) was created in Tolkien's Legendarium, and the details of that are provided in the Ainulindalë, The Music of the Ainur.

[In the beginning of creation] There was Eru, the One, who in Arda is called Ilúvatar; and he made first the Ainur, the Holy Ones, that were the offspring of his thought, and they were with him before aught else was made.
_{The Silmarillion, Ainulindalë}

But now Ilúvatar sat and hearkened, and for a great while it seemed good to him, for in the music there were no flaws.
_ibid.

The Music of the Ainur begins as a perfect harmony, as Eru wished, and would in turn create a perfect world. Within the host of the Ainur, however, was a powerful being named Melkor. He was the most powerful of the Ainur and tried to instil his own thoughts on the Music.

But as the theme progressed, it came into the heart of Melkor to interweave matters of his own imagining that were not in accord with the theme of Ilúvatar; for he sought therein to increase the power and glory of the part assigned to himself. To Melkor among the Ainur had been given the greatest gifts of power and knowledge, and he had a share in all the gifts of his brethren.
_ibid.

Throughout the building of the world Melkor continues to corrupt the works of the other Ainur and it continues from his influence on the Music to affect the paths of time.

and [the Valar] built lands and Melkor destroyed them; valleys they delved and Melkor raised them up; mountains they carved and Melkor threw them down; seas they hollowed and Melkor spilled them; and naught might have peace or come to lasting growth, for as surely as the Valar began a labour so would Melkor undo it or corrupt it...
<ibid.>

It is the corruption of Melkor that continues throughout the history of Arda that you comment on being "horrible". However there is some hope.

Arda Healed

In the Silmarillion is mentioned a second Music during which the themes of Ilúvatar shall be played "aright":

... though it has been said that a greater still shall be made before Ilúvatar by the choirs of the Ainur and the Children of Ilúvatar after the end of days. Then the themes of Ilúvatar shall be played aright, and take Being in the moment of their utterance, for all shall then understand fully his intent in their part, “and each shall know the comprehension of each, and Ilúvatar shall give to their thoughts the secret fire, being well pleased.
_ibid.

From the suggestions of the Ainulindalë, at the end of days (Dagor Dagorath) there shall be a new music that takes form immediately and reflects the will of Ilúvatar perfectly. A form of Utopia.

Finrod suggests that this third Arda will be the same as Arda Unmarred but greater:

'This then, I propound, was the errand of Men, not the followers, but the heirs and fulfillers of all: to heal the Marring of Arda, already foreshadowed before their devising; and to do more, as agents of the magnificence of Eru: to enlarge the Music and surpass the Vision of the World!'

'For that Arda Healed shall not be Arda Unmarred, but a third thing and a greater, and yet the same.'
_{Morgoth's Ring}

As well as Manwë:

"The second is the Unmarred that shall be: that is, to speak according to time in which they have their being, the Arda Healed, which shall be greater and more fair than the first, because of the Marring: this is the Hope that sustaineth."
_ibid.

Answer 2

One could argue, from an ontological perspective, that Eru (God) allowed evil to exist in the first place, in order to ultimately allow an example to be set for the others not to do evil.

Free will (or the illusion thereof) is the greatest gift given to the living. Like a kind father, Eru does not destroy Melkor when this latter first shows his true nature. Instead he suffers Melkor's continued existence and all that will follow as a result of this mercy. By allowing evil to continue to exist until its downfall, he shows – and remember, there is no better lesson than a direct example – that such things will not sustain themselves in the future: Evil consumes itself.

If the very existence of free will and choice allows tyranny to emerge as a consequence of those choices, per se, then the greatest good would be the opposite of that – utterly non-tyrannical – as in Eru's case, who shows mercy by creating, and trusting to others (each of whom are individually of lesser powers than Melkor) to join together in the fight against tyranny.

The flipside of this is that by showing mercy to one (greater) soul, he abandons many others to the lasting misery that this tyranny will inflict on them, c.f. the fate of elves who are turned into orcs, and ents into trolls. However, without seeing the full plan, we cannot know if there is a bliss of eternity that awaits them after their release from this mortal coil. (We know only what is written of the afterlife in the Silmarillion, which differs by race, and JRRT's own cultural heritage.)

(Is it better to exist and to suffer, or to not exist at all? Anti-natalism ponders this question, today.)

This is a much deeper and much older question that has little to do with Tolkien, specifically, and has ties to real world questions of good and evil.

Answer 3

Curiously, in The Two Towers there is a discussion between Frodo and Sam about the difference between stories which are good to read and stories which are good to experience as characters in them.

And the answer to why there is so much horror and sffering, death and destruction, in Middle-earth is:

Because Eru is a story teller like Tolkien. Only Eru tells stories with the fates of living beings instead of fictional characters.

In "Where No Man Has Gone Before" Captain James Kirk said "Above all else, a god needs compassion." And I have to suspect that Eru might possibly lack compassion, since Eru is supposedly all powerful. If Eru is all powerful, Eru should be able to heal Arda (the world) with a single thought of command, but He doesn't. Thus I have to suspect that if Eru is all powerful He is less than all knowing and all good.

In the real world Tolkien began writing stories that were the first versions of stories in the Silmarillion back during World War One, and kept on revising them for fifty or sixty years. And Tolkien began thinking about the philosophical basis of his fictional world decades after the main parts of the plots were firmly established in his mind.

So it was no longer possible for Tolkien to rewrite his stories so that they would be good for the characters to experience. In my opinion, the best story for any fictional characters to experience would be something like "Once upon a time, the universe was created. And everyone lived happily ever after.".

But if Tolkien ever came to that conclusion it was too late to rewrite his stories to make them more like "Once upon a time, the universe was created. And everyone lived happily ever after.".

And possibly Eru never came to the conclusion that it was wrong to tell intensely dramatic stories at the expense of the real people who lived, suffered, and died in them. Which you might find especially frightening due to the events in Tolkien's stories allegedly happening on Earth before recorded history. Tolkien said that The lord of the Rings happened about six or eight thousand years ago, for example. Thus in the works of Tolkien Eru is still today, in the year AD 2020, in charge of everyone's destiny and still telling exciting stories at the expense of endless horror, suffering, and death.

And if any real god or gods should read this, I hope that they will change the plots of future events to make future history closer to "Once upon a time, the universe was created. And everyone lived happily ever after.".

KIRK: Did you hear him joke about compassion? Above all else, a god needs compassion.

http://www.chakoteya.net/StarTrek/2.htm[1]