Inheriting the magical blood down the line, via a wizard father or a witch mother is understandable in case of Half-Bloods.
I am wondering as to how Muggle-borns end up with magical ability?
The best theory I can suggest is that magical ability is like a recessive gene. If each Muggle parent is a carrier, they can produce a magical child. (However, this theory only works if you exclude Squibs :P)
Also, quoting JK Rowling:
"Muggle-borns will have a witch or wizard somewhere on their family tree, in some cases many, many generations back. The gene resurfaces in some unexpected places."
A study in the British Medical Journal concluded that while there is a genetic basis for magical ability it involved the interaction of several genes, some recessive. This explains why muggles can produce magical offspring.
The study addressed not just general magical ability but specific skills such as the ability to speak to snakes.
The study is here: http://www.bmj.com/content/335/7633/1305.full.pdf
As ykombinator wrote, JK Rowling has spoken on this issue herself.
I believe it is likely that there's a series of genes tied to having magical ability (plus some magic, I'm sure, since there simply CAN'T be a purely biological explanation).
It's probable that there are a series of genes that are linked together, and that many Muggles have 'damaged' versions of these genes - this could account for the various mediums, psychics, etc that exist in the Muggle world (those who aren't faking it, at least). If this is the case, a single random mutation in the gene could fix it for two muggle parents, or two muggles could have a child who gets healthy versions of the gene from one or more parents (the genes do seem to be dominant, since Muggle + Wizard invariably ends up with a Squib or a witch/wizard).
Some of the genes likely control the ability to see magical creatures (these must be fairly robust, since Squibs can see Dementors), some to the ability to control magic, etc. Squibs most likely come in various degrees of magical ability (or non-ability) as there's at least the perception that they can learn magic - otherwise the market for Squib magic teaching supplies would dry up quickly.
It could be:
a random alignment of junk DNA that winds up coding for magic sensitivity, as was stated before
that magic chooses and unborn/young child somehow linking with them
proximity to magic in use in utero or in early childhood
random chance (such as the existence of squibs in major wizard families)
Take your pick, I pulled the last three out of a hat. They all are based on random chance, and a small one at that.
Magical tallent, if it is indeed genetic, would be linked to a wide number of genes. Many of the most famous witches and wizzards in the harry potter series are folk heroes who are well known as half human. Halfbloods not of wizarding blood and civilians, but of normal humans and some variety of magical creature. Merlin himself was a demigod, son of the celtic godess of magic and one of her last mortal worshipers, a woman with no talent for the art. Cercie, babayaga, Morgan LeFay and many others, hell they even included angreboda, the mother goddess of the frost giants durring the 4th book.
Add to that the interview with JKR herself who said that there's really no such thing as muggleborns, but rather the multigenerational decendants of squibs the answer seems clear to me. You want magic ability you have to have some sort of magical ancestry.
As a magician and muggle, I think magic happens out of intention. Most of the great masters of the art talk about the need to believe and intend the magic to happen and it will. I believe this is true in both fantasy and reality.
One of the lessons of the book, illustrated by Hermionie, is that people rely too heavily on blood and genes to define their talents. People labeled as Squibs are still capable of basic magic it just doesn't come as easily to them.
I would wager that anyone is capable of becoming a wizard if given the time, discipline, tools and knowledge. Just like people are not just born with a given talent for math, people pick up traits through their life and growth. One who grows up in a wizard family will likely pick up wizarding skills, just like someone growing up in a house of mathmaticians probably picks up math pretty easily.
Nature vs. Nurture.
Most of these types of questions assume that magical people have some special bit of DNA that turns them magical. In other words:
However, that only works when you look at humanity in isolation. When we take a look at the whole HP universe, we have to take into account that non-magical species outnumber magical species. Why? Additionally, we have to take into account that magical humans are only a small percentage of the human species, which doesn't make sense if magic was in any way beneficial, because evolution would have selected against non-magical variations in that case.
The only logical conclusion therefore is that magic isn't a beneficial ability. Which makes sense when you start looking at possible diseases. When you're magical, your body has to be able to defend itself against both magical and non-magical diseases. However, when you're non-magical, your body only has to defend itself against non-magical diseases.
In a way, the ability to perform magic is like colour-blindness. It's a disadvantage for the individual, but it can be beneficial to the population as a whole as long as it occurs only in a small percentage of the population.
So it should be the other way around:
I believe that squibs are the reason for muggle-borns. Centuries ago, I would imagine that squibs would have been cast out of wizarding society. Those squibs would have mingled with the muggle population. Say, generations down the line, two different squibs decendents marry and have a child. That child would have a greater chance to produce magic because his/her genetical makeup still has the magical gene, though stronger than either of his/her's parents. I do wonder however if the child of a regular muggle and the decendent of a squib would be magical.