Based on an answer on Ilvermorny school's sorting ceremony, multiple houses of the school can express an interest in a student, and some students have received a request for all four houses.
Have any students ever not been chosen by an Ilvermorny house, or any wizarding schools? Have there been any Hatstalls to the point of rejection once arriving at a school?
Neither at Hogwarts or Ilvermorny is there any mention of a student who is actually supposed to be attending the school but isn’t chosen for any house. Harry does worry that he wouldn’t be chosen for any house, but there’s nothing implying that it’s actually a possibility.
“A horrible thought struck Harry, as horrible thoughts always do when you’re very nervous. What if he wasn’t chosen at all? What if he just sat there with the hat over his eyes for ages, until Professor McGonagall jerked it off his head and said there had obviously been a mistake and he’d better get back on the train?”
- Harry Potter and the Philosopher's Stone, Chapter 7 (The Sorting Hat)
This seems likely to be Harry’s own worries rather than something that actually would happen at Hogwarts, and he didn’t know about Ilvermorny so it wouldn’t be applicable there.
In Hogwarts, it seems likely that any student who doesn’t really fit into any of the houses would go to Hufflepuff, as Helga Hufflepuff was willing to take all students into her house.
“Said Hufflepuff, ‘I’ll teach the lot,
And treat them just the same.”
- Harry Potter and the Order of the Phoenix, Chapter 11 (The Sorting Hat’s New Song)
The Sorting Hat was intended to continue choosing the students for each house in the same way the Founders did, and Helga Hufflepuff was willing to take the students not chosen by any of the others. Presumably, then, students who don’t fit anywhere would still go to Hufflepuff.
“Good Hufflepuff, she took the rest,
And taught them all she knew,
Thus the houses and their founders
Retained friendships firm and true.”*
- Harry Potter and the Order of the Phoenix, Chapter 11 (The Sorting Hat’s New Song)
Because of this, it’s highly unlikely that any students at Hogwarts wouldn’t be Sorted into a house, as if they don’t have any of the four houses’ traits, they’d likely be just put into Hufflepuff.
It’s a bit more difficult to deduce exactly what would happen with a student in Ilvermorny who’s ill-suited to all four houses. There’s no house in Ilvermorny that clearly would be willing to take the students who don’t really fit in anywhere. There’s nothing that indicates that there’s any case where no carving lights up to take a student, however, so there may be something that ensures at least one of them will ‘choose’ them even if they’re not a good fit just so all students have a house. In Ilvermorny, they also have inter-house competitions, so any students not being placed in a house would complicate matters as they’d have no house to compete for.
The house expanded with every passing year. More students had arrived and while the school was still small, there were enough children to fulfil Webster’s dream of inter-house competitions.
- Ilvermorny School of Witchcraft and Wizardry (Pottermore)
Though it’s unclear exactly what it is, there’s presumably some way that Ilvermorny ensures all of its students are accepted into one of the houses, even if no house really ‘fits’ them. As Valorum suggests in the comments, it’s possible that the student then gets to choose, like they would if they were chosen to be in more than one of the Ilvermorny houses.
The schools other than Hogwarts and Ilvermorny aren’t said to have houses. Since the other known wizarding schools don’t seem to have houses, there’d be no chance that a student wouldn’t fit into a house - all students would simply attend the school and none of them would have houses.
Using the term student liberally we have the case of Angus Buchanan, a squib. He was never actually a true student of Hogwarts, in that he never got the letter, but he did manage to get as far as the Sorting Hat so was a student in that he got to Hogwarts. Once in the school the Hat revealed for all that he was a squib and so he left. Again it isn't entirely clear if he was formally rejected from the school after never having been formally a student or if he left of his own accord.
By the mid-nineteenth century the Buchanan family comprised an overworked mother, a fierce father and eleven children. The household was loud and chaotic, but even so, it is surprising that neither of the Buchanan parents realised that their third son, Angus, was a Squib – a wizard-born child with no magical powers. [...]
His brothers and sisters were all very fond of Angus, who was the largest and kindest of them all, so they covered up for him in front of his parents. The deception was innocently begun, but as the time approached for him to leave for Hogwarts, Angus and his siblings became uneasily aware that they could not maintain the pretence much longer. No letter from school arrived for Angus, but his panicking sister Flora forged one, which kept the parents in ignorance for several weeks more. Shy, good-natured and frightened of his father, Angus could not think of any alternative but to play along with his older siblings. They took him to Diagon Alley, where they bought a wand and pretended that it had chosen him. On the appointed day, his big brother Hamish took him to Hogwarts on the back of his broomstick, hoping against desperate hope that Angus would be allowed to stay once they got there, or that the school might be able to tease some magic out of him.
It had never happened before and it has never happened since, but Angus got as far as the Sorting Hat before he was exposed. In sheer desperation he threw himself ahead of a girl whose name had been called and placed the Hat upon his head. The horror of the moment when the Hat announced kindly that the boy beneath it was a good-hearted chap, but no wizard, would never be forgotten by those who witnessed it. Angus took off the hat and left the hall with tears streaming down his face.
News of Angus’s humiliation reached his parents in a flurry of owls before their son arrived home on foot. He was met by his humiliated father, who barred his entrance, bade him never darken their door again, and fired curses after Angus as he fled.
Another unclear statement is the following "It had never happened before and it has never happened since". It could mean either a squib has never gotten as far as the Hat before being exposed or the Hat has never rejected anyone before. If the latter then this is the best example we have of a student not having been sorted before... at Hogwarts that is.
