It's worth noting that parental abandonment in some form or other is an extremely common trope in juvenile literature and particularly prevelant in British literature. Whether due to literal abandonment, neglect (often to the extent that would be considered abuse these days), boarding school, displacement during war or, ultimately, death.
It's a theme in most of Dahl's books, as well as many of the books listed as influences on Rowlings in the article linked in the previous answer. I'd consider it as the Harry Potter books being part of a broad literary tradition rather than it being an explicit reference to any specific book.
Some well known examples:
Death: Harry Potter, The Witches, James and the Giant Peach
Abandonment: The Chronicles of Prydain
Wartime Displacement: The Chronicles of Narnia, The Lord of the Flies
Neglect (by contemporary standards, not making a judgement here it can just be a literary device): Enid Blyton, E Nesbit
Boarding School: Tom Brown's Schooldays, Billy Bunter, Saint Trinians
Of course this theme can also be found in literature from other countries, intended for adults and feature prominantly in many fairy tales. It's just particularly prevalent in British children's literature (at a guess I'd say it was due to abandonment issues due to displacement during WWII and the prevalence of boarding schools).