Dudley could hurt Harry, so Harry wasn't totally safe at his Aunts house. First, were the Dursleys the only muggles that could harm Harry? Second, could the Deatheaters have simply defeated the charm protecting 4 Privet Drive by hiring muggle thugs to get him during summer break?
Harry had no protection from muggles. Any of them could have harmed him in any way that they wished (except that Harry might have fought back).
However, the Death Eaters had nothing but contempt for muggles. They would not have hired a muggle for any reason, but especially not to kill Harry, since Voldemort wanted to do that himself.