When I was 17, I had heard a couple stories about a magical place in Cleveland where one could pay a woman a small amount of money (not small to ME, but to fellows who bought $40 ounces of pot, it was small) and she would perform like a wife on your honeymoon hopefully would.
I knew about the whole thing, I knew where it happened, I knew how to make it happen, if there. But I had no way to get there and probably wouldn't have spent that kind of money on it even if I did have a way. I could honestly have said I've never left my county of my own volition, yet... yet... I still KNEW of it, and if presented with two lads I knew to be... scamps... right in front of me, with me now, by separate chance happening to be in the physical area of that magical place, seeing their huge grins and held-up-by-a-hand pants... I could have made the necessary connections and said what Sam said (well, something analogous). Never been there before, but fully aware of the underlying situation, and very able to make a reasonable (or unreasonable, who cares?) guess as to what had transpired.
So why can't Sam have the same (analogous, that is) situation? No continuity break.
But when my wife writes, this is the kind of thing I bring up (if I notice it) during editing. I tell her to maybe rethink how to lay some thing out, and maybe only the language used, because it might cause a reader to "stumble" on it while he works it through (or maybe doesn't work it out and starts to think you're just a bad writer) and then restarts his reading. Even if he does, he might not find the stumbling pleasant. But that's a different thing from continuity breaking. Practical problem, not a careless theoretical problem.
(And frankly, I'd be willing to spot Tolkien one or two here and there, if teensy like this one would be if it were so. A whole world, a whole universe, is a big thing to keep track of. I'm glad it wasn't much of an issue for him though!)