Sorry, I am not satisfied with the answers given here to that question, because Tolkien knew exactly the answer to that question. It has 2 levels. First the rational one for Thorin: in "Unfinished talesUnfinished Tales - the quest"The Quest to Erebor", Gandalf insists on this:
"Take a hobbit with you! Smaug has probably never heard of hobbits and he has certainly never smelt them." That is the reason Gandalf gave to Thorin..Thorin….
Then, there is the metaphysical one: Gandalf was the wisest of the Maïar and was specifically chosen by Manwë for the Istari mission because of that (see "Unfinished tales Unfinished Tales - the"The Istari"). Also of course, like all ValarsValar and MaïarsMaïar, having been one of the singers of the music of Illuvatar at the beginning of time he knew much of what would happen in Arda throughout time. And even if in his human form as a wizard sent to Middle-Earth by the Valar he didn't have the full memory of these things and his full Maïar powers, he was still the wisest of them and somehow knew in his heart that the Hobbits would play a great role in the events to come. In the "SilmarillonThe Silmarillion - Of"Of the ringRings of powerPower and the third age"Third Age", he said this to Elrond after a council of the wise that happened before the quest to Erebor:
"Many are the strange chances of the world, and help oft shall come from the hands of the weekweak when the wise falter".
And that is the reason why he insisted so much on the dwarves taking Bilbo with them. He didn't really knew the why, but his instinct told him the how. That is even why he was interested in that quest to begin with. Else, he wouldn't even have bothered about dwarves wanting their gold back, that was of no interest to him, that is quite obvious from his attitude with them.
