First, keep in mind that when The Hobbit was published, it was essentially a standalone story. Tolkien had the idea of hobbits independently of his already well-developed but unpublished work on what is now known as the First Age. In modern terms, we'd say that The Lord of the Rings is a giant retcon. The now-common text of The Hobbit is itself a revised edition of the 1937 original, made to fit the sequel better.
Appendix A of The Lord of the Rings includes the story Durin's Folk. This story is the canonical answer to your question. It briefly tells the tale of the Dwarves' time in Erebor before and after the coming of Smaug, the longing for a reconquest of Khazad-dûm (Moria), and the fate of Thráin (Thorin's father). It also gives an account of Gandalf's meeting with Thorin. Gandalf wanted Smaug destroyed in preparation for a full-fledged war against Sauron, in order for the Dwarves in the Iron Hills not to have to fight on two fronts.
Among many cares [Gandalf] was troubled in mind by the perilous state of the North; because he knew then already that Sauron was plotting war, and intended, as soon as he felt strong enough, to attack Rivendell. But to resist any attempt from the East to regain the lands of Angmar and the northern passes in the mountains there were now only the Dwarves of the Iron Hills. And beyond them lay the desolation of the Dragon. The Dragon Sauron might use with terrible effect. How then could the end of Smaug be achieved?
Tolkien started to write a more detailed story of the background to The Hobbit, but he never finished it. It is published in the Unfinished Tales under the title The Quest of Erebor.
After Gandalf's meeting with Thorin, he was unconvinced by the latter's plan to fight Smaug directly. He thought the Dwarves didn't have a chance, and would sacrifice their lives in a foolish venture. As Gandalf happened to visit the Shire next, he thought of sending a hobbit along, to contribute stealth to the party.
Suddenly in my mind these three things came together: the great Dragon with his lust, and his keen hearing and scent; the sturdy heavy-booted Dwarves with their old burning grudge; and the quick, soft-footed Hobbit, sick at heart (I guessed) for a sight of the wide world.
Gandalf further realizes that he has obtained from the dying Thráin in Sauron's jail a way to enter Erebor unnoticed (the map and key). It is not clear from that version if Gandalf has an actual premonition that Bilbo's participation is necessary, or if this is merely a realistic evaluation of the party's chances and an argument to convince Thorin to take Bilbo on. In Gandalf's words:
I knew in my heart that Bilbo must go with him, or the whole quest would be a failure. (…) ‘Listen to me, Thorin Oakenshield !’ I said. ‘If this hobbit goes with you, you
will succeed. If not, you will fail. A foresight is on me, and I am warning