Wizard wars
Potters vs. Dursleys. Both Dursley parents are alive. Both Potter parents are dead, killed by other wizards. So while wizards may be able to survive specific threats better, in practice, they survive less often. Because someone like Voldemort doesn't regard muggles as worth killing unless they have some personal relationship. It is other wizards that they find threatening.
It's not just the Potters. The Longbottoms are another example. Not dead, but not reproducing either. And doubtless there are many other dead wizards that we don't know because they died off camera without progeny.
Magical threats
Wizards also face magical threats that muggles don't. Sure, a muggle is less able to face a dragon, but in practice, the wizards put themselves between the muggles and the dragon. Because the wizards don't want the much more numerous muggles finding out about magic.
Beyond all that, the wizards know how to find dragons and can use magic to go places that muggles cannot. Wizards face more dangers than muggles.
The Weasleys are an exception
Yes, a wizarding family can have more kids. Just as first world countries could have more kids than third world countries. But in practice, we see the reverse. Most families in rich countries (Japan, Europe) only have one or two kids, below replacement on average. While poor countries have large families. Muggles have historically had more kids than wizards.
Yes, the Weasleys are a big family. But many other wizards are only children, e.g. Neville Longbottom, Scorpius Malfoy (so far), Draco Malfoy, and Lucius Malfoy. Some wizarding couples have no children, e.g. Bellatrix and Rudolphus Lestrange. And not all wizards marry, e.g. Albus Dumbledore and his siblings (or their aunt Honoria), Sirius Black, Radastan Lestrange, and Barty Crouch, Jr. (another only child).
We don't have statistics on wizarding families as far as I know. So this tends to produce a lot of anecdotes. Is only child Harry an exception or common? The way that wizards act, he seems more common than not.
