It starts with a man, his wife, his son and his son's girlfriend in bunker. The wife and son defect to the aliens, the man ends up with his son's girlfriend, and fights the aliens.
If it's Farnham's Freehold, then the "aliens" are simply negroes who have somehow taken over the post-apocalyptic world. The surviving white Americans are slaves. The housekeeper is accepted into the "Chosen" because he is black, but the son ends up being castrated. When they get an opportunity to return to their own time, he refuses to go with them in his present condition. So both he and the housekeeper may be said to have defected.
It is revealed toward the end that the Chosen are cannibals, which ruffled a few feathers and led to mutterings about "racism" from the politically correct.