Harry’s grandparents, on both sides of his family, were dead.
Both sets of Harry’s grandparents were dead, which J.K. Rowling explicitly said was done for the plot. She also gives the backstory behind their deaths - Lily’s parents died a normal Muggle death, and James’s were both old and died of a wizarding illness.
MA: What about Harry's family — his grandparents — were they killed?
JKR: No. This takes us into more mundane territory. As a writer, it was more interesting, plot-wise, if Harry was completely alone. So I rather ruthlessly disposed of his entire family apart from Aunt Petunia. I mean, James and Lily are massively important to the plot, of course, but the grandparents? No. And, because I do like my backstory: Petunia and Lily's parents, normal Muggle death. James's parents were elderly, were getting on a little when he was born, which explains the only child, very pampered, had-him-late-in-life-so-he's-an-extra-treasure, as often happens, I think. They were old in wizarding terms, and they died. They succumbed to a wizarding illness. That's as far as it goes. There's nothing serious or sinister about those deaths. I just needed them out of the way so I killed them.
- The Leaky Cauldron (July 16, 2005)
They were all dead, specifically because Rowling wanted them out of the way and Harry to have no family other than Aunt Petunia so the plot would have more reason to go the way it did.