Mass Effect 3 does not properly explain this, so we have to speculate to some extent.
In general, a population must have a certain minimum size in order to be viable in the long run. Otherwise, excessive inbreeding is required to continue reproducing at the replacement rate. A straightforward answer to this question would be that the vast majority of quarians did leave Rannoch, and the small number left behind were unable to maintain a stable population for the 300 year period between the Morning War and the events of the game. This explanation has the advantage of being relatively straightforward, not requiring too many assumptions, and not depending on how the quarians and geth interacted with each other after the Morning War.
As another point, by this stage in their evolution, the quarians were not all living on Rannoch in the first place. They had colonized a number of planets, according to Tali in Mass Effect, and the order to shut down the geth "went out" simultaneously in all of these places. It's possible that the quarians were actively and deliberately migrating people from Rannoch to their colonies, perhaps in order to prop up the legitimacy of their claimed territories, but also scattering their population, and making it less able to recover from the sudden loss of such a large number of people all at once.
Finally, the question arises of how these renegade quarians are feeding themselves. Their population has just gotten dramatically smaller, and the geth are no longer providing them with slave labor. Somebody has to go out and till the fields, and for the time being, the quarians are probably doing that by hand. Perhaps the geth would choose to provide them with volunteer labor, as some sort of gesture of goodwill, but we see no evidence of this in Mass Effect 3. As a result, they might struggle to reproduce at the replacement rate anyway, due to scarcity. It's even possible that they lost the practical ability to engage in interstellar trade due to a severe reduction in specialized labor (a prerequisite for any kind of space travel).
Out of universe, it's obvious that Bioware did not want to create a "maskless quarian" game model; they even show masked quarians in the flashback quest, where they make little logical sense. If the player pursues a romantic relationship with Tali, she provides an image of herself without her mask, but it's just a photoshopped stock photo.* This problem has impacted Mass Effect before:
- We never see female turians before Mass Effect 3: Omega.
- The first time we see an elcor walking, it's from a great distance in Mass Effect 2, and only in a scripted cutscene.
- Urdnot Bakara wears a thick set of veils, making it difficult to discern what she actually looks like. For all we know, her model is otherwise identical to, say, Grunt's. We never encounter any un-veiled female krogan until Andromeda.
* In Mass Effect: Legendary Edition, this image has been replaced, and to my eye, the new image looks like it was made from scratch. But it's still just a 2D image, not a 3D in-game model.
