In the novels, Vader is working to get Luke to fall, as is Palpatine, but Vader wants Luke to swear loyalty to him, and not Palpatine.
The first goal is to get Luke to actually commit himself to the Dark Side; righteous anger is too easily quenched by reason. Luke could have killed Palpatine there and then, and then gone right back to the Light leaving Vader alone, or even turned on Vader after killing Palpatine. Palpatine said it himself: "he has grown strong; only together can we turn him towards the Dark Side of the Force". So, Vader had to defend Palpatine because the first goal had not yet been met; Luke had to see that hate was power, feel it coursing through his veins, in order to give himself to the Dark Side. He wasn't there yet when he first struck at Palpatine.
However, by defending the Emperor and thus forcing Luke to fight him, Vader took the focus off Palpatine. He was too good at getting Luke riled up, especially when he discovered Luke had a twin sister and used that to provoke him. Luke forgot all about the Emperor and focused all his hate on Vader, and it wasn't until the Emperor's laughter jarred Luke out of his battle haze that he realized just how close he was to falling and stepped back.
Vader, at that point, had lost completely. Luke wouldn't turn, and if he had it would have been at Vader's cost. Vader really had no choice but to crawl back to his master's side. Except... if Palpatine won, the Skywalkers would die; Palpatine had already shown that he considered Vader to be just another pawn, to be sacrificed for his own ends. Luke would die on the Death Star, and Palpatine wouldn't stop until Leia was dead too (and she was on Endor; in the novels, if the shield ever came down, Jerjerrod was under orders to destroy Endor). Vader had loved Padme even after falling, and seeing their children suffer, watching one being tortured to death and knowing the other was right behind, was just too much. He found another option, the one he never would have considered under any other circumstances; destroy Palpatine himself, turn on his master, end the Sith, to save not only his son but his daughter as well.