Magic is an inherited trait, genes likely do affect it in some ways.
Magical ability itself is an inherited trait - it’s very rare that children born to at least one wizard parent won’t be born as wizards themselves, and instead will be Squibs with no magical powers.
“A Squib is someone who was born into a wizarding family but hasn’t got any magic powers. Kind of the opposite of Muggle-born wizards, but Squibs are quite unusual.”
- Harry Potter and the Chamber of Secrets, Chapter 9 (The Writing on the Wall)
Muggle-borns also inherit their magical ability, but from further back in their ancestry.
“As intensive studies in the Department of Mysteries demonstrated as far back as 1672, wizards and witches are born, not created. While the “rogue” ability to perform magic sometimes appears in those of apparent non-magical descent (though several later studies have suggested that there will have been a witch or wizard somewhere on the family tree), Muggles cannot perform magic.”
- The Tales of Beedle the Bard
Therefore, ancestry affects magical ability in at least one provable way - whether it’s present at all.
Ancestry does seem to affect chances of having wizard children.
Wizards continuing to only marry wizards does seem effective at ensuring that further generations would also be wizards - the Black family tree shows they’ve been consistently mostly wizards since the Middle Ages. Squibs would have been blasted off the tree and their descendants not shown, so all those shown on the Black family tree would be from a bloodline of wizards, and especially those on the lower branches would have only had wizard ancestry going back generations. As the Black family tree isn’t largely made up of burn marks, it seems they have a low rate of producing Squibs.
“The tapestry looked immensely old; it was faded and looked as though Doxys had gnawed it in places. Nevertheless, the golden thread with which it was embroidered still glinted brightly enough to show them a sprawling family tree dating back (as far as Harry could tell) to the Middle Ages. Large words at the very top of the tapestry read
The Noble and Most Ancient House of Black
‘Toujours pur”
- Harry Potter and the Order of the Phoenix, Chapter 6 (The Noble and Most Ancient House of Black)
It does seem like intermarrying consistently with Muggles will make wizards less likely to produce magical descendants, as the Scourers, a group of wizard mercenaries, hid by marrying Muggles and winnowing out magical offspring for non-magical. If they consistently producing magical offspring at the rate that at least one magical parent would, it’d be incredibly difficult for them to make any children they could keep even after generations, as Squibs (non-magical children born to at least one magical parent) are rare. Too much intermarriage likely can turn the bloodline Muggle.
Several of the most notorious Scourers eluded justice. With international warrants out for their arrest, they vanished permanently into the No-Maj community. Some of them married No-Majs and founded families where magical children appear to have been winnowed out in favour of non-magical offspring, to maintain the Scourer’s cover.
- Seventeenth Century and Beyond (Pottermore)
However, marrying too close within the same family will result in certain complications, like instability, weakness, and other problems. The Gaunts we know of were indeed all wizards but they were also all obviously unstable and deformed due to their interbreeding.
“Marvolo, his son Morfin and his daughter Merope were the last of the Gaunts, a very ancient wizarding family noted for a vein of instability and violence that flourished through the generations due to their habit of marrying their own cousins.”
- Harry Potter and the Half-Blood Prince, Chapter 10 (The House of Gaunt)
There doesn’t seem to be any similar complications from marrying only wizards for generations as long as they’re far enough apart in relation - marrying Muggles specifically isn’t shown to be necessary to avoid these once there’s enough genetic distance between those choosing to marry.
There’s no clear evidence how much or if ancestry affects talent.
Once magical ability is inherited, it’s unlikely how much having magical ancestry affects the talent of wizards. There may or may not be any inherited component to how skilled a wizard is or can become, especially considering that magical ability is in fact inherited. Hagrid thinks that because Harry’s parents were, but it’s unclear if this is based in fact or his own optimism regarding Harry.
“A wizard, o’ course,’ said Hagrid, sitting back down on the sofa, which groaned and sank even lower, ‘an’ a thumpin’ good’un, I’d say, once yeh’ve been trained up a bit. With a mum an’ dad like yours, what else would yeh be?”
- Harry Potter and the Philosopher's Stone, Chapter 4 (The Keeper of the Keys)
Whether wizards without magical ancestry are generally equally as powerful as wizards with magical ancestry is even less clear. Many statements on this by individual wizards in the books are obviously biased in one direction or another, either for or against those with Muggle ancestry, so finding the truth on if or how much magical ancestry affects a wizard’s talent or potential talent is quite difficult. There aren’t many statements in the books themselves that can be considered objective on this, and JKR doesn’t seem to have said anything about whether magical ancestry affects magical talent. It also isn’t easily discernible from the wizards seen in the books and what’s known of their skills and blood status - it’s too small a sample size to get accurate results from figuring out how magically talented the wizards we know are and cross-referencing that with their blood status to see which status produced the most skilled wizards.
