This is definitely hard to explain. If the Wizarding community interbred with muggles for thousands of years, population genetics would definitely lead us to expect that most humans alive today would be magical, since everyone would have at least some magical ancestors.
I get the impression from the books that interbreeding was uncommon in the past, even though it's now common. That would make it possible for most people to still not have any magical ancestors. The rise of interbreeding means that will change in the future, and we should expect a large increase in the percentage of the population with Wizarding abilities.
The facts about heritability of Wizarding, as I understand it:
- Most children of two Wizard parents will be magical. (And no I'm not going to write Witch-or-Wizard everywhere. Please assume Wizard is a unisex term like Muggle.)
- Most children of Wizard + Muggle will be magical
- One Muggle parent doesn't make Squib offspring much (or at all?) more likely than for two wizard parents. (This is the most difficult part to explain with any kind of simple Mendelian genetic model.)
- Muggle-born witches and wizards are descended from Squibs who married Muggles; the magical ability unexpectedly resurfaces after many generations. (source).
A google search for harry potter population genetics
turned up a high-school science lesson plan on heritability of genetic traits using Harry Potter characters and creatures as example, published by the U.S. National Library of Medicine. The end of the second lesson (pdf) proposes a theory of heritability of magical ability where magical ability is not expressed in Muggles because they carry at least one copy of a silencing version of a gene (allele) which regulates the activity of the allele for strength of magical ability.
Their theory explains some of the phenomena from the books, but not the low rate of Squibs / Muggles from Wizard + Muggle parents. (According to their theory, every Muggle has at least one copy of the genetically dominant silencing allele, so even in the best case, children from ss(wizard) + sS(muggle) have a 50% chance of inheriting an S (silencing gene) and being pure Muggle, with not even weak magical ability. Some Muggles would carry an SS genome and be incapable of ever producing magical offspring, so clearly this model doesn't match this aspect.
Traits caused by multiple alleles have more complex behaviour, but I suspect there might not be any purely Mendelian explanation. I think the fact that children of Wizard + Muggle are no less likely to be Squibs is the hardest thing to explain. Perhaps this is a case of unreliable narrators, since the characters saying this are fighting a war against pure-blood supremacists! (And out-of-universe, I think JKR didn't want her story to include a genetic basis for discrimination.)
Perhaps at this point, interbreeding is still a new thing, and Wizards almost always have two copies of the magical allele(s). Perhaps after some generations of mixing, Muggle + Wizard will have a significantly lower chance to produce magical offspring than Wizard + Wizard.
Or maybe it's magic. Fortunately, invoking magic to explain magic is not exactly a problem, but it means we can't just use formulae from evolutionary biology / population genetics.
The extreme heritability of Wizarding should make it spread much more easily / strongly than a normal genetic mutation. It will easily become fixed throughout a population as soon as a few Wizards are common ancestors of all living humans. (This happens after enough generations, even with limited interbreeding.)
The most recent common ancestor (MRCA) of all humans alive today may have lived as recently as a few thousand years ago. The existence of long-isolated tribes may push that number back as far as the Upper Paleolithic, but a few thousand years or even shorter is probable if we don't worry about isolated pockets and just consider the bulk of Western civilization (the setting for Harry Potter).
At some point before the time of the MRCA, there's a point where everyone in the population is either an ancestor of everyone alive today or has no living descendants.
However, if Wizards limit their interbreeding with Muggles, the populations could stay separate. Wizardlyness can't spread throughout the Muggle population because of the practice of removing Wizards from their Muggle parents and bringing them into the Wizarding world. At that point they no longer count as Muggles, and are more likely to marry (or whatever) other Wizards.
The un-expressed form (that leads to Muggle-born wizards) could spread throughout the Muggle population, but IDK what the implications of that are. IDK if it's normal that Muggle-borns can identify a specific Squib ancestor, or if some just happen from random chance (i.e. inherited from an ancient magical ancestor many tens of generations ago, i.e. magic is just circulating in the gene pool.)