Because that was the shortest and least dangerous route to go
Just to clarify: Gandalf wasn't going by another path to Erebor by himself, he was heading to Dol Guldur first with The White Council , then afterwards rejoining Bilbo and the Dwarves. The other way round that Gandalf mentions would take the party more time to travel, in addition to being more dangerous. Bilbo himself asks Gandalf this.
"Do we really have to go through?" groaned the hobbit.
"Yes, you do!" said the wizard, "if you want to get to the other side. You must either go through or give up your quest. And I am not going to allow you to back out now, Mr. Baggins. I am ashamed of you for thinking of it. You have got to look after all these dwarves for me," he laughed. "No! no!" said Bilbo. "I didn't mean that. I meant, is there no way round?"
"There is, if you care to go two hundred miles or so out of your way north, and twice that south. But you wouldn't get a safe path even then. There are no safe paths in this part of the world. Remember you are over the Edge of the Wild now, and in for all sorts of fun wherever you go. Before you could get round Mirkwood in the North you would be right among the slopes of the Grey Mountains, and they are simply stiff with goblins, hobgoblins, and orcs of the worst description. Before you could get round it in the South, you would get into the land of the Necromancer; and even you. Bilbo, won't need me to tell you tales of that black sorcerer. I don't advise you to go anywhere near the places overlooked by his dark tower! Stick to the forest-track, keep your spirits up, hope for the best, and with a tremendous slice of luck you may come out one day and see the Long Marshes lying below you, and beyond them, high in the East, the Lonely Mountain where dear old Smaug lives, though I hope he is not expecting you."
The Hobbit, Queer Lodgings
Cutting straight through Mirkwood on the Old Forest Road would take far less time. That does not mean it wasn't dangerous, as Beorn himself notes that goblins used that road.
But Beorn had warned them that that way was now often used by the goblins, while the forest-road itself, he had heard, was overgrown and disused at the eastern end and led to impassable marshes where the paths had long been lost.
The Hobbit, Queer Lodgings
You can see for yourself on this map: To go North? An extra 200 miles and guaranteed goblins at the end? No thank you. 400 extra miles South and going near Dol Guldur? I'll pass. Going through Mirkwood on the Old Forest Road; goblins, possibly, but it would sure take a ton of time off the journey.
So their last alternative, the Elf-path and also their route of choice, seemed the most ideal because, like the Old Forest Road, it was a shortcut through Mirkwood to Erebor. However unlike the latter it was a lesser-known path, meaning the chances of meeting goblins were much less on that path than any other. (Thanks @chepner)
North of the Carrock the edge of Mirkwood drew closer to the borders of the Great River, and though here the Mountains too drew down nearer, Beorn advised them to take this way; for at a place a few days' ride due north of the Carrock was the gate of a little-known pathway through Mirkwood that led almost straight towards the Lonely Mountain.
The Hobbit, Queer Lodgings
Map of Wilderland
The map of Wilderland drawn by Christopher Tolkien and included in most editions of The Hobbit.
As you can see on the above map the Elf-Path is located a little more North as compared to the Old Forest Road, and would clearly take the party far less time travelling on it and through the forest, than going either North or South to avoid Mirkwood.