At that point in the story Voldemort had just been reborn a couple of hours earlier. For the previous 13 years he had not been around, and the dementors had been working for the Ministry. In the graveyard Voldemort claimed that the dementors would join them, but that is a future reference:
The dementors will join us... they are our natural allies...
Similarly, Dumbledore argued that the dementors would join Voldemort, again speaking of the future:
"Voldemort has returned," Dumbledore repeated. "If you accept that fact straightaway. Fudge, and take the necessary measures, we may still be able to save the situation. The first and most essential step is to remove Azkaban from the control of the dementors -"
"Preposterous!" shouted Fudge again. "Remove the dementors? I'd be kicked out of office for suggesting it! Half of us only feel safe in our beds at night because we know the dementors are standing guard at Azkaban!"
"The rest of us sleep less soundly in our beds, Cornelius, knowing that you have put Lord Voldemort's most dangerous supporters in the care of creatures who will join him the instant he asks them!" said Dumbledore. "They will not remain loyal to you, Fudge! Voldemort can offer them much more scope for their powers and their pleasures than you can! With the dementors behind him, and his old supporters returned to him, you will be hard-pressed to stop him regaining the sort of power he had thirteen years ago!"
When Harry gets attacked by dementors that summer, he wonders whether the dementors had already left the Ministry's control, once again implying that at the time of the attack on Crouch Jr. the dementors had not yet left the Ministry's control:
But at these words of Uncle Vernon’s, Harry’s
exhausted brain ground back into action. Why had
the dementors come to Little Whinging? How could it
be coincidence that they had arrived in the alleyway
where Harry was? Had they been sent? Had the
Ministry of Magic lost control of the dementors, had
they deserted Azkaban and joined Voldemort, as
Dumbledore had predicted they would?
Even after that, at Harry's hearing, Fudge still maintained that the dementors were under Ministry control rather than Voldemort's:
“It means that I think they were ordered there,” said
Dumbledore.
“I think we might have a record of it if someone had
ordered a pair of dementors to go strolling through
Little Whinging!” barked Fudge.
“Not if the dementors are taking orders from someone
other than the Ministry of Magic these days,” said
Dumbledore calmly. “I have already given you my
views on this matter, Cornelius.”
“Yes, you have,” said Fudge forcefully, “and I have no
reason to believe that your views are anything other
than bilge, Dumbledore. The dementors remain in
place in Azkaban and are doing everything we ask
them to.”
“There are no dementors outside Ministry control!”
snapped Fudge, who had turned brick red.
And Dumbledore's evidence to the contrary was that dementors had been ordered to Harry's neighborhood, but that evidence was undermined when we found out that it was actually Umbridge who ordered the dementors to attack Harry:
“What Cornelius doesn’t know won’t hurt him,” said
Umbridge, who was now panting slightly as she
pointed her wand at different parts of Harry’s body in
turn, apparently trying to decide what would hurt the
most. “He never knew I ordered dementors after
Potter last summer, but he was delighted to be given
the chance to expel him, all the same...”
In short, the dementors were not loyal to Voldemort at the time that Crouch got his soul sucked out.
Additionally, even when the dementors are technically on a specific side, they don't really care who they harm. Note Dumbledore's words of caution when the dementors were stationed at Hogwarts:
"It is not in the nature of a dementor to understand pleading or excuses. I therefore warn each and every one of you to give them no reason to harm you. I look to the prefects, and our new Head Boy and Girl, to make sure that no student runs afoul of the dementors," he said.