It is said that split souls can be repaired, as long as its owner has not yet died. The catch is that the process can itself be lethal, and it requires the owner to feel sincere remorse: not something that someone vile enough to create a Horcrux is ever likely to do.
I bring this up because I suspect that it's crucial to finding an answer to your question. I would imagine that un-Horcruxed fragments do whatever they would have done if they'd never been Horcruxed in the first place, so that repairs remain possible, but we don't know what "whatever they would have done" is. Further details of the repair process could give us clues to that. Does it involve actually placing the fragments back into the soul in some way? Does the soul "call back" its fragments, which then rejoin with it on their own? Or does the soul "grow back" its lost pieces, with the original fragments not even needed?
If we assume the first case -repairing a soul means putting the fragments back in- then the fragments probably stay "with" the soul, in whatever way the concept of location applies to souls and fragments. They can't actually rejoin until repairs are made, but we never hear anyone talk about needing to find lost fragments, so if the repair process needs to do something with them, then finding them must be too easy to be worth mentioning. Un-Horcruxed segments, then, would likely return to wherever the soul is, even though they can't actually rejoin with it.
In the second case -the soul calls its fragments back and mends itself- then the fragments must still exist somewhere, but their location isn't important. Wherever they went, they'd hear the call, which would explain why they don't need to be found. Unless fragments have some ability to move about in some way (other than being called back), then they'd probably stay right where they were, waiting to be called. If they could move about, then they would.
In the last case -lost pieces grow back when the soul is repaired- then fragments probably just dissipate soon after being split off. They're not needed anymore, which again explains why they don't need to be found. Un-Horcruxed fragments, then, would probably dissipate in the same way.