Uh, sheer numbers and time.
Ego has to return to his home planet from time to time in order to replenish his power, and he's probably got to be there to build and perhaps maintain his planet.
We don't know how many women Ego slept with and where they were in the galaxy relative to his home base. We also don't know that he stopped sleeping with women to continue this after he made Peter.
My impression was that Ego had so many that Yondu not coming back with Peter wasn't something that deeply troubled him, and he had a lot of other kids coming in from different sources or he went and got the closer ones himself (see KSmarts answer--really do think other folks were ferrying them in for Ego). Only once he heard about the Infinity Stone, did Ego decide to look, and put together that it had to be Peter. If he investigated at all, Peter Quill's name would be in any report regarding this. And Ego would know who that is, just from the last name.
Basically, with all the boning, the breaking in of children, the maintaining of his planet and power, dude just didn't have the time to pick up all his kids when they got "ripe." Earth may be geographically further away than some other planets, so it's possible Ego picked some of his own kids up.