We see there in The Hobbit itself.
First, we've got this:
Indeed Bilbo found he had lost more than spoons—he had lost his reputation.
It is true that for ever after he remained an elf-friend, and had the honour of dwarves, wizards, and all such folk as ever passed that way; but he was no longer quite respectable. He was in fact held by all the hobbits of the neighbourhood to be ‘queer’—except by his nephews and nieces on the Took side, but even they were not encouraged in their friendship by their elders.
This quote implies that these folk did occasionally pass by.
And indeed, right after that, who comes by?
One autumn evening some years afterwards Bilbo was sitting in his study writing his memoirs—he thought of calling them “There
and Back Again, a Hobbit’s Holiday”—when there was a ring at the door. It was Gandalf and a dwarf; and the dwarf was actually Balin.
The Hobbit, chapter 19: "The Last Stage"
And later on in The Fellowship:
But the Gaffer did not convince his audience. The legend of Bilbo’s wealth was now too firmly fixed in the minds of the younger generation of hobbits.
"Ah, but he has likely enough been adding to what he brought at first," argued the miller, voicing common opinion. "He’s often away from home. And look at the outlandish folk that visit him: dwarves coming at night, and that old wandering conjuror, Gandalf, and all. You can say what you like, Gaffer, but Bag End’s a queer place, and its folk are queerer."
The Lord of the Rings: The Fellowship of the Ring, chapter 1: "A Long-Expected Party"