Malfoy let out a terrible scream and bolted – so did Fang. The hooded figure raised its head and looked right at Harry – unicorn blood was dribbling down its front. It got to its feet and came swiftly towards him – he couldn’t move for fear.
Then a pain pierced his head like he’d never felt before, it was as though his scar was on fire – half-blinded, he staggered backwards. He heard hooves behind him, galloping, and something jumped clean over him, charging at the figure.
The pain in Harry’s head was so bad he fell to his knees. It took a minute or two to pass. When he looked up, the figure had gone. A centaur was standing over him.
Philosopher's Stone - page 187 - Bloomsbury - chapter 15, The Forbidden Forest
At the time Harry encountered Voldemort in the Forbidden Forest, as Quirrell drank the unicorn's blood, Quirrell was probably at, or close to, his magically strongest, and he also the benefit of Voldemort's powers (although they are weakened).
Centaurs, on the other hand, don't seem to have overt magical abilities -- they read the skies and rely on the bow and arrow as weapons.
So how is it that Firenze, without magic, was able to drive away Quirrell/Voldemort? It would seem that Quirrell/Voldemort could have easily incapacitated Firenze with magic and gone after Harry.
Why was Quirrell scared away by Firenze?