the problems are even more complicated in regards to Tolkien's Middle-earth/Arda related material
The answers above do a great job. But I'd like to elaborate on one complication that was only alluded to. The books actually exist inside the fictional universe. Even if you care about canonicity (if, OP wonders, that's a word), still, fictional historiography (yes, that's a word) might be important to you.
The Lord of the Rings (as stated explicitly, within itself: this is canon!) is derived from the Red Book of Westmarch, written by the hobbits: Bilbo, then Frodo, and then Sam, with help from Pippin and Merry. They were eyewitnesses to most of the events it tells, and they had friendship with surviving eyewitnesses to almost all of the rest. Almost.
Gollum looked at them [Frodo and Sam sleeping]. A strange expression passed over his lean, hungry face. The gleam faded from his eyes, and they went dim and grey, old and tired. A spasm of pain seemed to twist him, and he turned away, peering back up towards the pass, shaking his head, as if engaged in some interior debate. Then he came back, and slowly putting out a trembling hand, very cautiously he touched Frodo’s knee – but almost the touch was a caress. For a fleeting moment, could one of the sleepers have seen him, they would have thought that they beheld an old weary hobbit, shrunken by the years that had carried him far beyond his time, beyond friends and kin, and the fields and streams of his youth, an old starved pitiable thing.
The Two Towers: The Stairs of Cirith Ungol
(Hobbitses knows so much, even sleeping, yes precious, even though Poor Smeagol burned, yes, and didn't keep journals, did he precious, nassty scribbling hobbitses, we hates them...)
Since there were no surviving witnesses to this part, and since an "internally-consistent canon" must include the fact that the book exists in-universe, I can still ask you whether this scene is for "real", or whether perhaps it is an embellishment by Frodo, or by a later editor. Isn't that a perfectly valid in-universe question?
To answer it, you might point to some letter from Tolkien (this one perhaps). This letter, you say, is canon, so this part actually happened. Then again, maybe you don't consider letters to be canon, or consider them a different level of canon, or something. Then this scene would have a different canonical status from most of the rest of the book.
This example is more of a nitpick. To be honest, all this is mostly an excuse to quote one of my favourite passages! But I hope it illustrates that even a "fan perspective" can still recognize the difference between whether something "canonically was written" and whether it "canonically happened". For the Lord of the Rings, there are only minor differences. But what about the Silmarillion?
The Silmarillion itself does not state anything about its own source, as a whole. There are statements here and there about parts of it. Also, it has a foreword by Christopher Tolkien, quoted in one of the above answers, and I'll requote:
... my father came to conceive The Silmarillion as a compilation, a compendious narrative, made long afterwards from sources of great diversity (poems, and annals, and oral tales) that had survived in agelong tradition ...
You might ponder what the canonical status of his conception is.
Indeed, one of the definitions quoted above is that, ideally, canon is what "Tolkien would have eventually published" had he lived. "We don’t and never will know what the ideal canon looks like." Very well. Still, I think we do know that Tolkien would have eventually taken care to place the Silmarillion itself inside the universe, one way or another. Besides his son's testimony, let me quote:
But the chief importance of Findegil's copy is that it alone contains the whole of Bilbo's "Translations from the Elvish". These three volumes were found to be a work of great skill and learning in which, between 1403 and 1418, he had used all the sources available to him in Rivendell, both living and written. But since they were little used by Frodo, being almost entirely concerned with the Elder Days, no more is said of them here.
Fellowship of the Ring: Prologue
Perhaps Tolkien would have eventually written that the Silmarillion was one, two, or three of these books. Or parts thereof. Or something else, who knows, but whatever the case, it's very likely that the Silmarillion is meant to have in-universe existence in some form, like the Lord of the Rings. Except the latter was basically written by eyewitnesses, whereas the in-universe counterpart(s) to the Silmarillion would have been condensed from a variety of sources, having very different points of view, and not always contemporary, or reliable.
On the one hand, Elrond was at the sack of Balar, Galadriel crossed the Grinding Ice, and Glorfindel was at the fall of Gondolin. (Or was he? Determining whether two people with the same name are actually the same person is just the sort of thing real historians have to deal with.) But other stories would have to be second-hand, third-hand, or less. (And who interviewed Morgoth and Ungoliant?)
Also consider Tolkien's "retcon" of the Hobbit, which was alluded to above.
‘Don’t do that!’ said Gandalf, sitting down. ‘Do be careful of that ring, Frodo! In fact, it is partly about that that I have come to say a last word.’
‘Well, what about it?’
‘What do you know already?’
‘Only what Bilbo told me. I have heard his story: how he found it, and how he used it: on his journey, I mean.’
‘Which story, I wonder,’ said Gandalf.
‘Oh, not what he told the dwarves and put in his book,’ said Frodo. ‘He told me the true story soon after I came to live here. He said you had pestered him till he told you, so I had better know too. “No secrets between us, Frodo,” he said; “but they are not to go any further. It’s mine anyway.”’
‘That’s interesting,’ said Gandalf. ‘Well, what did you think of it all?’
‘If you mean, inventing all that about a “present”, well, I thought the true story much more likely, and I couldn’t see the point of altering it all. It was very unlike Bilbo to do so, anyway; and I thought it rather odd.’
Fellowship of the Ring: a Long-Expected Party
Tolkien not only put the books inside the universe, he also took care to put the retcon inside the universe: Bilbo was the author of the first version, which was a distortion motivated by the One Ring, and he told the true version to Frodo. What's more: in-universe, Frodo and Gandalf know the contradicting stories, and examine the two versions critically. This, too, is canon!
So even if you start out looking for "canon", in the end, a desire for internal consistency might lead you to think like a fictional historian anyway.