Hogwarts is modeled on English private schools like Eton. If Hogwarts is a "private school" where do the students go who either can't afford it or don't get in?
What if they can’t afford to go?
I don’t recall any mention of school fees in the books. With regard to school supplies, see Dumbledore’s visit to a young Tom Riddle:
“That is easily remedied,” said Dumbledore, drawing a leather money-pouch from his pocket. “There is a fund at Hogwarts for those who require assistance to buy books and robes. You might have to buy some of your spellbooks and so on secondhand, but—”
— Half-Blood Prince, chapter 13 (The Secret Riddle)
JK Rowling later confirmed on Twitter that Hogwarts has no tuition fee:
There's no tuition fee! The Ministry of Magic covers the cost of all magical education!
What if they don’t get in?
As for students who “don’t get in” or choose not to go, their parents can educate them at home or abroad, if they so wish. We learn that these are possible when Lupin explains the changes at Hogwarts in the final book:
“Attendance is now compulsory for every young witch and wizard,” he replied. “That was announced yesterday. It’s a change, because it was never obligatory before. Of course, nearly every witch and wizard in Britain has been educated at Hogwarts, but their parents had the right to teach them at home or send them abroad if they preferred.”
— Deathly Hallows, chapter 11 (The Bribe)
However, it should be noted that anybody who doesn’t “get in” may not have much to gain. Part of Pottermore entitled The Quill of Acceptance and the Book of Admittance describe how students are entered for the school. If you trust the Quill and Book as accurate (as the article seems to do), then it explains that it keeps out people without any magic:
In fact, the Book's sternness has a purpose: its track record in keeping Squibs out of Hogwarts is perfect. Non-magic children born to witches and wizards occasionally have some small, residual aura of magic about them due to their parents, but once their parents magic has worn off them it becomes clear that they will never have the ability to perform spells. The Quill's sensitivity, coupled with the Book's implacability, have never yet made a mistake.
If you assume that it really can list every magical child in Britain, then anybody who doesn’t get in is either a Muggle or a Squib, and so won’t really benefit from education elsewhere.
What do you mean, "don't get in" ?
There are no mentions in books of any fees or admission process that would cause a young British wizard or witch to be turned away from Hogwarts. You could assume that they exist, but that's just fanfic-topic with no basis in canon.
Being poor does cause difficulties in obtaining study materials and tools, but it doesn't prevent them from going to Hogwarts as such, as evidenced by multiple such students shown in the books.
All the other answers are explaining about options for people who (or whose parents) choose not to go to Hogwarts; but if you're asking about those who "don't get in" then that's limited to squibs - and they can either be homeschooled or go to any muggle school they like.
They either get in on some sort of charity or scholarship, I mean that is how Voldemort got in. Or they obviously go to another school, like Durmstrang or Beauxbaton. People like Hermione probably got in partly on scholarship or charity, as well as Ron. Can you imagine her parents Dentist or not, trying to come up with gold to pay for her schooling,they would have went bankrupt. Also, people that were poor like Ron got in, so there was obviously some sort of charity. Also, people were homeschooling their children. Remember how in the 7th book of Harry Potter the ministry was trying to get all of the students registered or something like that, so that they would know who was muggleborn or not. Also the students could do what Grindlewald did, after he was expelled from Durmstrang he went traveling, although this would only be ideal for older students since younger aren't good enough in magic, yet.
This if it is canon might help you. The source was sketchy, though.
Do all young people in the Wizarding World (from Britain) go to Hogwarts? For example, did Stan Shunpike attend Hogwarts? Or is Hogwarts a school just for those who are particularly good at magic while others go into trades without formal schooling? Jo's response: (all magical young people are invited, not all attend) This is another example of an answer that tells us nothing at all. She didn't actually say if Stan attended, for example. She didn't give us any hint of how many kids who have magical ability choose not to attend. So we're left with no answer to the basic question. Is Hogwarts an elite school, attended only by those who aspire to something more than working in a trade? There's a bit of an inconsistency in Jo's answer, too. She says they're no level to magic, that either you have it or you don't. But some characters are clearly noted as having more magical ability than others. Barty Crouch Sr. for example is described as "powerfully magical." I think she intends that statement to mean that there's no minimum level of magic which allows you to go to Hogwarts, below which you have to be a dishwasher in the Leaky Cauldron or something like that. http://www.hp-lexicon.org/jkr-letter-responses.html
This is from the wikia.
Hogwarts is considered to be one of the finest magical institutions in the wizarding world, though other notable schools include Beauxbatons Academy of Magic and the Durmstrang Institute. Children with magical abilities may be enrolled at birth, and acceptance is confirmed by owl post at age eleven.
So my official response on it is that a lot of students have the ability to go to the school, but not all go obviously. I just remembered after seeing the link the @alexwchan posted in a comment that JKR in Pottermore did say that they were chosen by a magical quill, so the students that after being chosen or not being chosen that decided not to go to Hogwarts would have to find another school or be home schooled. I forgot where it was said but I heard that Mrs. Weasley homeschooled the Weasleys before they went to school, she was a stay at home mom after all.
Hogwarts is not an exclusive institution! Every single magical children born in the British Isles immediately gets a place in Hogwarts. There's no acceptance requirements or anything. If you're magical, you get in. We don't know if students pay tuition—my guess is not—but as shown by Riddle, Hogwarts has a bursary for poor students, so that shouldn't be a reason not to attend it. In short, everyone who wants to go to Hogwarts can do so.
What about the people who don't want to? Well, they can be either homeschooled, or sent to another magical school overseas.
Students can be homeschooled, or go to a foreign school.
Draco mentioned that he wanted to go to Durmstrang, so it is possible.
Presumably if they don't get into Hogwarts they attend one of the other magical schools OR they integrate into muggle school if they don't have any magical ability (squib).