The primary evidence for the assumption that the Silmarillion was Bilbo's three volumes comes from the prologue to Lord of the Rings:
But the chief importance of Findegil's copy is that it alone contains the whole of Bilbo's 'Translations from the Elvish'. These three volumes were found to be a work of great skill and learning in which, between 1403 and 1418, he had used all the sources available to him in Rivendell, both living and written. But since they were little used by Frodo, being almost entirely concerned with the Elder Days, no more is said of them here.
It's well-documented that there are plenty of changes that Christoper Tolkien regrets making to the published Silmarillion, but one change he regrets not making is given in the foreword to Book of Lost Tales 1:
So also I have assumed: the 'books of lore' that Bilbo gave to Frodo provided in the end the solution: they were 'The Silmarillion'. But apart from the evidence cited here, there is, so far as I know, no other statement on this matter anywhere in my father's writings; and (wrongly, as I think now) I was reluctant to step into the breach and make definite what I only surmised.
While the original framing device of a mariner becoming lost off the western shores of Europe and finding the ancient Elvish lands, where he's told the stories, gradually fell away it was never wholly discarded, but was removed from the published Silmarillion by Christopher Tolkien. Again from Lost Tales 1:
The letter of 1963 quoted above shows my father pondering the mode in which the legends of the Elder Days might be presented. The original mode ... had (by degrees) fallen away. When my father died in 1973 'The Silmarillion' was in a characteristic state of disarray: the earlier parts much revised or largely rewritten, the concluding parts still as he had left them some twenty years before; but in the latest writing there is no trace or suggestion of any 'device' or 'framework' in which it was to be set. I think that in the end he concluded that nothing would serve, and no more would be said beyond an explanation of how (within the imagined world) it came to be recorded.
This isn't entirely true and in fact the last versions of several of the tales still contain the old mode; for example, from CT's commentary on the Akallabeth (hoME 12):
But with the removal of Pengolod and Ælfwine from the published text, the Akallabeth lost its anchorage in expressly Eldarin lore; and this led me (with as I now think an excess of vigilance) to alter the end of the paragraph.
Tolkien being Tolkien, this of course was by no means consistent and while this framing device remained in some of the stories, it was entirely absent from others, while others were presented as works of lore by the Eldar of Tol Eressea, and others again were given without any context at all.
The final word is left to Christopher Tolkien, in his introduction to the Silmarillion:
Moreover, my father came to conceive The Silmarillion as a compilation, a compendious narrative, made long afterwards from sources of great diversity (poems, and annals, and oral
tales) that had survived in agelong tradition.
And that's where we must accept it.