The task, as described as "Collect the Golden Egg", is misleading - that's not all there is to the task.
Borrowing From @BeginTheBeguine's answer here about why the other contestants weren't really discussed much:
"You were the best, you know, no competition. Cedric did this weird thing where he Transfigured a rock on the ground. . . turned it into a dog. . . he was trying to make the dragon go for the dog instead of him. Well, it was a pretty cool bit of Transfiguration, and it sort of worked, because he did get the egg, but he got burned as well - the dragon changed its mind halfway through and decided it would rather have him than the Labrador; he only just got away. And that Fleur girl tried this sort of charm, I think she was trying to put it into a trance - well, that kind of worked too, it went all sleepy, but then it snored, and this great jet of flame shot out, and her skirt caught fire - she put it out with a bit of water out of her wand. And Krum - you won't believe this, but he didn't even think of flying! He was probably the best after you, though. Hit it with some sort of spell right in the eye. Only thing is, it went trampling around in agony and squashed half the real eggs - they took marks off for that, he wasn't supposed to do any damage to them."
Harry Potter and the Goblet of Fire - Chapter 20: "The First Task"
(Emphasis mine)
Or in short - the Task is not just to get the Golden Egg - it's to safely collect the Golden Egg, and get safely it and yourself away from the Dragon.
Accio'ing the Golden Egg just between the Dragon's legs would have been risky, as the Dragon would then be able to respond once it sees it going through their legs, or feels it moving, and react. If it destroys the egg, you have no chance to complete the task, and if it destroys you, then there's nowhere for the Golden Egg to be Accio'd to.
The point of Harry going with the broomstick result he went with is that he is good at flying, and was hoping to use the speed at which he can move to get away from the dragon - with the egg also with him effectively acting like a Quidditch Snitch for his Seeker skills -, and be out of range of any retaliation.
Furthermore, as quoted in @The Dark Lord's answer, Charlie revealed to Hagrid that the task didn't just have dangerous dragons - they had, specifically, nesting mother dragons.
While Accio'ing the broom takes some time to show up, the dragon can't, until it shows up into sight, intercept it and ruin it. Plus, had the dragon somehow intercepted the broom before Harry could get hold of it, Harry could, hypothetically, just summon another broom. It wouldn't be great for him, perhaps, but it would be repeatable.