Being in the air makes one a more visible target than if they were on the ground. A greater amount of enemies can see you and falling could be fatal. It's harder to protect yourself from behind if you fly past someone and leave your back undefended. On the ground, you can approach an opponent head on without having to worry about blind spots as much.
During the battle on brooms, from Deathly Hallows Chapter 23:
A Death Eater moved ahead of Snape and raised his wand, pointing it directly at Lupin's back- Sectumsemra! shouted Snape. But the spell, intended for the Death Eater's wand hand, missed and hit George instead.
Granted, Snape maybe wasn't the best of flyers, but he still managed to miss using his own signature spell. Not everyone can hold their wands steady on a broom or even fly well at all. During the same battle in Deathly Hallows Chapter 4, Hagrid uses his motorcycle's magic to slow down enemies in pursuit:
Three of the Death swerved and avoided it, but the fourth was not so lucky: he vanished from view and then dropped like a boulder from behind it, his broomstick broken into pieces. One of his fellows slowed up to save him.
And again shortly after, two Death Eaters must briefly leave the battle after Harry uses the Confringo curse:
The Death Eater nearest it was blasted off his broom and fell from sight; his companion fell back and vanished.
That's twice that combatants had to pause to rescue a falling ally, while in a normal duel the injured fighter might not have been in mortal peril. During a duel or a battle, it's safer to stay on the ground rather than become a more visible target or risk falling to one's death- or falling into the enemy's ranks. Additionally, broomsticks are expensive and few people could bankroll their allies like that.