With Silmarillion, Tolkien wanted to create a mythology similar to for example the Norse one, and that's what he considered his most important work. He wanted Silmarillion to be belivable, and so he didn't just produce the stories, but languages, maps, family trees, pantheons, races etc etc.
The stories in The Hobbit and LotR weren't really planned to be integrated with the Silmarillion work initially. It wasn't until the massive success of LotR, when Tolkien needed to explain details of things in the books to the fans, that he fully integrated the LotR world with Silmarillion.
And so for example we get the legend of Gondolin briefly appearing in The Hobbit when they find the swords, mostly as a minor curiosity then - Tolkien was at that point writing a book for children and he didn't really intend it to merge it with the Silmarillion work then, the tale of Gondolin being one of his earliest works.
There are some legends appearing by chance like that, with Tolkien taking inspiration from his own work and not really explaining the legend in detail to the reader. Other such examples is the legend of Eärendil and the legend of Beren and Luthien, both appearing in LotR. To some of the characters (like the hobbits) the stories would be just that - legends. While others like Aragorn and Arwen would also know them as factual history of their own ancestors. But with the events of the First Age happening many thousand years before the LotR story, they would be a mix of history, legend and fairy-tales to the people living in the Third Age.
(Compare it to for example the stories of Ragnar Lodbrok and his sons in our world, where we lack records and aren't really certain what's fiction and what's history.)
Then there are various in-world legends or prophecies that Tolkien placed there on purpose for the plot. Prophecies are spoken by people (or Valar) with the gift of foretelling the future, which is fairly common throughout the books. In other cases such plot-central legends just exist as hearsay and tradition, such as the various legends surrounding the heir of Isildur: legends claiming that the King would return one day. Since these legends were present among the people of Gondor, Aragon's claim wasn't really questioned at all - it was sufficient that he brought the re-forged sword, the army of the dead and produced a sapling of the White Tree to prove that he was the King.
These kind of legends that Tolkien placed there on purpose are essential, I suppose, since they are important to the story. But they hold far less substance and details than the "historical legends" that entered his works more by coincidence. Take the legend/prophecy that said that the Witch King could be killed by no man - we have no idea where this originated from or why we should regard it as trustworthy - the legend is suddenly just there, dropped on top of the reader when the Witch King is introduced in detail for the first time.