When Harry first tells Hermione and Ron about the vision he had about Sirius in the Department of Mysteries, Hermione points out that it would be impossible since it's 5pm and the Ministry is full of workers. When the group gets to the Ministry, no one is there.
"But...Harry, think about this," said Hermione, taking a step toward him, "it's five o'clock in the afternoon...The Ministry of Magic must be full of workers....How would Voldemort and Sirius have got in without being seen? Harry...they're probably the two most wanted wizards in the world....You think they could just get into a building full of Aurors undetected?"
Order of the Phoenix - page 732 - Bloomsbury - chapter 32, Out of the Fire
I know that Voldemort or the Death Eaters somehow got everyone out, but how could they have possibly done that? There are hundreds are people working, including the Aurors and the Minister of Magic. And even if most people had gone home by that time, there still would have been security, etc.
Harry felt sure that there ought to be a security person there, sure that their absence was an ominous sign
Order of the Phoenix - page 769 - Bloomsbury - chapter 34, The Department of Mysteries
So if somehow Voldemort has the power to get everyone in the entire Ministry to evacuate, why didn't he just get the prophecy himself instead of using a likely to fail plan to get Harry to go? Bellatrix remarks about how Voldemort couldn't just walk through the Ministry, but why couldn't he? If it was completely empty? He shows up anyway at the end. And why is it more likely an underage wizard could get into the Department of Mysteries than a powerful grown wizard?
"Get it himself?" shrieked Bellatrix on a cackle of mad laughter. "The Dark Lord walk into the Ministry of Magic, when they are so sweetly ignoring his return? The Dark Lord, reveal himself to the Aurors, when at the moment they are wasting their time on my dead cousin?"
Order of the Phoenix - page 786 & 787 - Bloomsbury - chapter 35, Beyond the Veil
And if he used some enchantment to get everyone out, why did Fudge show up at the end? If there was some sort of alarm system that alerted Fudge, wouldn't it have gone off in the beginning? They were there for at least an hour before Fudge comes.