What would happen if someone rejected their Hogwarts Invitation Letter? Would the same thing that happened to Harry occur? With thousands of letters flying into his/her house?
They had the right to be homeschooled, or to study abroad.
As Lupin points out in, parents have the right to homeschool their children, or even send them abroad.
"Attendance is now compulsory for every young witch and wizard," he replied. "That was announced yesterday. It’s a change, because it was never obligatory before. Of course, nearly every witch and wizard in Britain has been educated at Hogwarts, but their parents had the right to teach them at home or send them abroad if they preferred."
Harry Potter and the Deathly Hallows, chapter 11 "The Bribe".
E.g. Lucius Malfoy considered sending Draco to Durmstrang:
". . . Father actually considered sending me to Durmstrang rather than Hogwarts, you know."
Harry Potter and the Goblet of Fire, chapter 11 "Aboard the Hogwarts Express"
And Ariana Dumbledore was said to be homeschooled due to her health, but we all know what really happened.
In fact, in most of countries homeschooling is the only, and the preferred, option:
The number of countries that have their own magical school is minuscule compared to those that do not. This is because the wizarding populations of most countries choose the option of home schooling.
Wizarding Schools on Pottermore (by J. K. Rowling)
As for Harry's case, it was very special, but we do know for a fact that Hogwarts sends special people to Muggleborn's families to brief them in on the Wizarding world (we thank @Valorum for finding this quote):
“And will it really come by owl?” Lily whispered.
“Normally,” said Snape. “But you’re Muggle-born, so someone from the school will have to come and explain to your parents.”
Harry Potter and the Deathly Hallows, chapter 33 "The Prince's Tale"
I would assume that most parents are not as stupid as Dursleys, and would prefer their child to be educated rather than roam the world with an uncontrolled magical power (see Ariana's case).
I don't know whether there's a canon indication of what would happen if someone turned down a place at Hogwarts, but I would imagine that, like any other school, Hogwarts would simply accept the decision and leave it at that. Harry wasn't pelted with letters because he said no, he was pelted with letters because he wasn't receiving them (and because, as @Tim noted in the comments, he was a special case, being the Boy Who Lived and all that).