Yes, it's very likely some attend Muggle schools before Hogwarts.
There's at least one case of a wizard born to Muggles who was very likely to have been attending a Muggle school before attending Hogwarts - Justin Finch-Fletchley, whose name had been down for Eton.
“My name was down for Eton, you know, I can’t tell you how glad I am I came here instead. Of course, mother was slightly disappointed, but since I made her read Lockhart’s books I think she’s begun to see how useful it’ll be to have a fully trained wizard in the family …” - Harry Potter and the Chamber of Secrets, Chapter 6 (Gilderoy Lockhart)
There are tests that prospective students have to pass before they'd be accepted to Eton, so for Justin to be accepted he'd have to have done sufficiently well on them. Homeschooling is legal in Britain, and it's possible that Justin could have been homeschooled instead of attending a school before he was due to start at Eton. However, the percentage of children who are homeschooled is small.
A spokesman for the Department of Children, Schools and Families cited a study done in 2006 in response to a question about the rising numbers of homeschooled children, which gives an idea how small the percentage likely is.
"These rather speculative findings must be understood in context. We are talking about very small rises of no more than a couple of hundred, set against the eight million children that are taught in mainstream education. In a study last year, the proportion of home educated children known in nine local authorities ranged from just 0.09 per cent to 0.42 per cent, so this practice is still extremely rare."
It would have been quite rare when Justin's parents would have decided how to educate him, so it's far more likely that he attended a school at some point.
In addition, Ernie Macmillan considered Justin saying he was going to be attending Eton if he hadn't instead gone to Hogwarts as a clear indication that he was born to Muggles, implying that Justin isn't unusual among wizards born to Muggles for having originally planned to attend a Muggle school.
“So anyway,’ a stout boy was saying, ‘I told Justin to hide up in our dormitory. I mean to say, if Potter’s marked him down as his next victim, it’s best if he keeps a low profile for a while. Of course, Justin’s been waiting for something like this to happen ever since he let slip to Potter he was Muggle-born. Justin actually told him he’d been down for Eton. That’s not the kind of thing you bandy about with Slytherin’s heir on the loose, is it?” - Harry Potter and the Chamber of Secrets, Chapter 11 (The Duelling Club)
British children are required by law to begin some form of education when they're five years old, long before they'd know their child was a wizard, since they only find that out when they receive their Hogwarts acceptance letter. It's very likely that most of them would have attended a Muggle school before that. Their parents wouldn't be keeping them out of school knowing they're a wizard, since they wouldn't know yet. While young wizards can cause odd things to happen around them, it's unlikely that this would put all parents off of sending their child to school, especially if there aren't many incidents. They also might not know or suspect it was magic and presumed there was a mundane explanation for any oddities.
Harry did also attend a Muggle school, but he would be a special case as his aunt and uncle knew he was a wizard since they got him but were actively trying to stamp the magic out of him. Sending him to a Muggle school could easily have been a part of that, so this in itself can't be considered representative of wizards born to Muggles in general.
“On the other hand, he’d got into terrible trouble for being found on the roof of the school kitchens. Dudley’s gang had been chasing him as usual when, as much to Harry’s surprise as anyone else’s, there he was sitting on the chimney. The Dursleys had received a very angry letter from Harry’s headmistress telling them Harry had been climbing school buildings.” - Harry Potter and the Philosopher's Stone, Chapter 2 (The Vanishing Glass)
Harry wasn't born to Muggles, he was known to be a wizard, and the Dursleys already knew they hated magic - they would likely have made him keep attending if Hagrid hadn't retrieved Harry, since he had a secondary school chosen for him even though his aunt would have known the age he'd be supposed to start Hogwarts.
“When September came he would be going off to secondary school and, for the first time in his life, he wouldn’t be with Dudley. Dudley had a place at Uncle Vernon’s old school, Smeltings. Piers Polkiss was going there, too. Harry, on the other hand, was going to Stonewall High, the local comprehensive.” - Harry Potter and the Philosopher's Stone, Chapter 3 (The Letters from No One)
Harry's case gives a bit of a look at what a wizard child going to Muggle school would be like, but his case shouldn't be considered the typical example of what a wizard born to Muggle parents would experience.
They wouldn't have to quit school, but wouldn't attend Muggle secondary school.
British Muggle children leave primary school and start secondary school at the same time that wizard children start Hogwarts. So, any wizard enrolled in Muggle primary school would be able to finish there. They would have graduated from their primary school by Hogwarts age. However, like Justin, they would no longer be able to attend any secondary school they might have already been on the waiting list for so they would have to cancel their enrollment for any Muggle secondary school they were enrolled in. To explain the child's absence from school, it's likely that their parents tell authorities they're going to homeschool their child, or possibly Hogwarts is registered as a secondary school with the government.