You are talking about an ontological paradox, and, in the show's terms, we don't know. Each time there has been the potential for one, something has occurred to prevent it. In the case of the episode I believe you are referring to, Rimmer goes and destroys the device that permitted the future group to exit, thus abrogating their future. True, he does this in response to them, but if looked at one way, his timeline can now proceed without interference, and they are merely a potential future that has been prevented.
Something similar exists in White Hole; at the end, with time going back to where it was, the whole episode never happened.. No paradox in this case, because the initiating events never happened, but our hero's are in the same position as the 'Future self' versions in this one; they are part of a time line that will now not happen, and fade from existence.. although Kryten does take the opportunity to take a jab at Rimmer, since it will never be seen.
And, then there is the episode with they cyborg that is judging all of mankind (one at a time), where Lister steals the time gauntlet and sets up a paradox to protect himself.. Which turns out to be useless, as (since he set up a situation where he saved the villain) were he deleted, he would never have set up the paradox in the first place... Although this is actually a gambit to sucker the bad-guy into zapping himself with the now rigged gauntlet, and take himself out of time.. again, reverting the entire cast to the pre-episode state.. In this case, though, we see alternatives of the crew, while Lister and Kryten are protected from deletion by the fact that the gauntlet has not yet deleted them from existence.
In each case, the actual paradox is prevented, resulting in the abrogation of a future time-line, so we really don't see WHAT would happen. The easiest way to look at it is to concentrate on the earliest (in a temporal sense) group; they are the ones that will continue, if possible. Think of them as the source of a stream or river. The future versions are possibilities, but, when the possibility is 'cut' by whatever action, that particular path for the river is blocked, and it finds a new path to follow. A true, self-destructive paradox (like their future selves killing themselves, and thus preventing their own existance, and, by extension, all of their actions) runs the river in a circle -- in real life, this would result in a pond, lake, etc, until it overflowed it's banks and found a new course -- in the Red Dwarf universe, I suspect that 'temporal potential' that would build up is what keeps such things from being possible.. In point of fact, they may have happened.. hundreds or thousands of times.. but they eliminate themselves, and repeat, until something changes.. all we can see is the results of the LAST change.
Oh, and then, of course, going back to early on, there are the 'Future Echos'; but these are an example of incomplete understanding of future events; no real paradox exists, as they are unable to change events thru not having a full understanding of what was going on.
The closest to an exception that I recall is Lister's parentage; that's a paradox, but it's not a self destructive one, it's simply the 'chicken and egg' type, in that we can't say how exactly it got started.
