Voldemort knew about the general location of Harry's relatives' house and the Ministry knew about the residents. The Ministry decided to go against Dumbledore at the end of Goblet of Fire and Voldemort already had influential people at the Ministry by that time, through Lucius Malfoy at least. Why wouldn't he try to attack any of the Dursleys outside No. 4 Privet Drive?
There is a saying that the Devil's greatest trick is in convincing people he does not exist. Common to many would-be tyrants is using denial and the status quo bias of the general public to paint ones opponents as crazy people with an ax to grind. This was how Voldemort's faction worked to suppress opposition to their agenda until it was too late to stop them. Dumbledore, Harry, Hermione, the Weasleys---All were supposedly nuts with an ax to grind. Right up until the day Voldemort and the Death Eaters took power, the evening of Bill and Fleur's wedding.
An attack on the Dursleys would have lent too much credence to the remaining Order of the Phoenix, and could have stopped the Death Eaters' rise to power at a critical juncture, plus given Harry enough warning to get out of their reach.