Because using a sword was effective.
(Keep in mind that Balrogs envisioned by Tolkien were not as big as what Peter Jackson depicts in the movies.)
'Do as I say!' said Gandalf fiercely. 'Swords are no more use here. Go!'
The Lord of the Rings: The Fellowship of the Ring, The Bridge of Khazad-dûm
In the books, Gandalf says that particular line before he is even aware that Durin's Bane is a Balrog. With hordes of Orcs and something bigger on the other side of the door, it would seem pointless for the Fellowship to not run away. Gandalf knew that only he could hold the door, with his magic, against those orcs. Not the Fellowship.
The passage was lit by no shaft and was utterly dark. They groped their way down a long flight of steps, and then looked back; but they could see nothing, except high above them the faint glimmer of the wizard's staff. He seemed to be still standing on guard by the closed door. Frodo breathed heavily and leaned against Sam, who put his arms about him. They stood peering up the stairs into the darkness. Frodo thought he could hear the voice of Gandalf above, muttering words that ran down the sloping roof with a sighing echo. He could not catch what was said. The walls seemed to be trembling. Every now and again the drum-beats throbbed and rolled: doom, doom.
The Lord of the Rings: The Fellowship of the Ring, The Bridge of Khazad-dûm
Gandalf had no idea that there was a Balrog until he was confronted by it.
Gandalf's sword: Glamdring
Also note that Glamdring was a legendary sword made in the First Age for King Turgon. It's easily the most ancient sword which still existed at the time of the Fellowship (since Narsil was made for King Elendil sometime in the Second Age). That doesn't necessarily mean it's the most "powerful", but the fact that it was made for the then High King of the Noldor means it could, at least, hold its own against a creature of Morgoth.
"This, Gandalf, was Glamdring, Foe-hammer that the king of Gondolin once wore."
The Hobbit
Durin's Bane vs Gandalf
It's also explicitly mentioned that Gandalf did use Glamdring against the Balrog.
'We fought far under the living earth, where time is not counted. Ever he clutched me, and ever I hewed him, till at last he fled into dark tunnels. They were not made by Durin's folk, Gimli son of Gloin. Far, far below the deepest delving of the Dwarves, the world is gnawed by nameless things. Even Sauron knows them not. They are older than he. Now I have walked there, but I will bring no report to darken the light of day.
The Lord of the Rings: The Two Towers, The White Rider
It did seem like it hurt the Balrog. And finally we have the 2 of them fighting it out at Zirakzigil.
A great smoke rose about us, vapour and steam. Ice fell like rain. I threw down my enemy, and he fell from the high place and broke the mountain-side where he smote it in his ruin. Then darkness took me; and I strayed out of thought and time, and I wandered far on roads that I will not tell.
The Lord of the Rings: The Two Towers, The White Rider
It's fairly unclear what Tolkien meant by "threw him down", whether by magic or a physical shove, but I see it as Gandalf, with Glamdring, pushing the Balrog back onto the edge of the peak that ultimately caused the Balrog to fall down.
Gandalf's use of magic
The strain of keeping the Balrog behind the door took its toll on Gandalf, as he explicitly states:
'A Balrog,' muttered Gandalf. 'Now I understand.' He faltered and leaned heavily on his staff. 'What an evil fortune! And I am already weary.'
The Lord of the Rings: The Fellowship of the Ring, The Bridge of Khazad-dûm
We do see him using his magic to stop the Balrog from crossing the bridge, but not any more after that. That doesn't mean he didn't use his magic, but I feel that he found his sword to be of more use against the Balrog than perhaps any spells he could cast. Remember, the Balrogs were once Maia as well, but corrupted. Gandalf too is a Maia, albeit clothed in human flesh that gave them restrictions on the power they could use in Middle-earth.