There were similarities in the plot lines and differences too.
Similarities
Sand snakes wanted revenge for their father's death:
“Tyene. Obara is too loud. Tyene is so sweet and gentle that no man will suspect her. Obara would make Oldtown our father’s funeral pyre, but I am not so greedy. Four lives will suffice for me. Lord Tywin’s golden twins, as payment for Elia’s children. The old lion, for Elia herself. And last of all the little king, for my father.”
A Feast for Crows, Chapter 2, The Captain Of The Guards
Oberyn Martell was popular in Dorne and the high born lords wanted Oberyn death to be avenged:
“To spears!” a man bellowed from a balcony.
“Doran!” called some highborn voice. “To the spears!”
Hotah gave up looking for the speakers; the press was too thick, and a third of them were shouting. “To spears! Vengeance for the Viper!”
A Feast for Crows, Chapter 2, The Captain Of The Guards
Doran was worried about the Sand Snakes:
“Give me your blessing, then, and I shall go.”
Doran hesitated half a heartbeat before placing his hand on his niece’s head. “Be brave, child.”
“Oh, how not? I am his daughter.”
“No sooner had she taken her leave than Maester Caleotte hurried to the dais. “My prince, she did not . . . here, let me see your hand.” He examined the palm first, then gently turned it upside down to sniff at the back of the prince’s fingers. “No, good. That is good. There are no scratches, so . . .”
A Feast for Crows, Chapter 2, The Captain Of The Guards
So in the story line of the TV show, the Sand Snakes and Ellaria Sand, use the unpopularity of Doran Martell to kill him and take over Dorne.
Differences
Different Plan:
In the books, Doran Martell has a daughter Arianne who plans to declare Myrcella Baratheon as Queen of the Seven Kingdoms, based on Dornish law - which allows an older sister to inherit before a younger brother - thereby precipitating a war with the Iron Throne and King Tommen Baratheon.
“I would be honored to arrange the wedding, and to see to the making of the crowns as well. Trystane and Myrcella are so innocent, I thought perhaps white gold . . . with emeralds, to match Myrcella’s eyes. Oh, diamonds and pearls would serve as well, so long as the children are wed and crowned. Then we need only hail Myrcella as the First of Her Name, Queen of the Andals, the Rhoynar, and the First Men, and lawful heir to the Seven Kingdoms of Westeros, and wait for the lions to come.”
“The lawful heir?” The prince snorted.
“She is older than her brother,” explained Tyene, as if he were some fool. “By law the Iron Throne should pass to her.”
“By Dornish law.”
A Feast for Crows, Chapter 2, The Captain Of The Guards
They had no intention of killing Trystane, Myrcella and Doran.
“Seven save us,” whispered Tyene. “Trystane? Why?”
“The woman must be mad,” Obara said. “He’s just a boy.”
“This is monstrous,” said Lady Nym. “I would not have believed it, not of a Kingsguard knight.”
“They are sworn to obey, just as my captain is,” the prince said. “I had my doubts as well, but you all saw how Ser Balon balked when I suggested that we go by sea. A ship would have disturbed all the queen’s arrangements.”
A Dance with Dragons, Chapter 38, The Watcher
Ellaria did not seek revenge:
“Oberyn wanted vengeance for Elia. Now the three of you want vengeance for him. I have four daughters, I remind you. Your sisters. My Elia is fourteen, almost a woman. Obella is twelve, on the brink of maidenhood. They worship you, as Dorea and Loreza worship them. If you should die, must El and Obella seek vengeance for you, then Dorea and Loree for them? Is that how it goes, round and round forever? I ask again, where does it end?” Ellaria Sand laid her hand on the Mountain’s head. “I saw your father die. Here is his killer. Can I take a skull to bed with me, to give me comfort in the night? Will it make me laugh, write me songs, care for me when I am old and sick?”
A Dance with Dragons, Chapter 38, The Watcher
Family is united:
Doran convinces his family to stay united and take on Lannisters.
“The Lannisters will not like it, no more than they liked it when I sent them Oberyn, but they dare not refuse. We need a voice in council, an ear at court. Be careful, though. King’s Landing is a pit of snakes.”
Lady Nym smiled. “Why, Uncle, I love snakes.”
“And what of me?” asked Tyene.
“Your mother was a septa. Oberyn once told me that she read to you in the cradle from the Seven-Pointed Star. I want you in King’s Landing too, but on the other hill. The Swords and the Stars have been re-formed, and this new High Septon is not the puppet that the others were. Try and get close to him.”
“Why not? White suits my coloring. I look so … pure.”
“Good,” the prince said, “good.” He hesitated. “If … if certain things should come to pass, I will send word to each of you. Things can change quickly in the game of thrones.”
“I know you will not fail us, cousins.” Arianne went to each of them in turn, took their hands, kissed them lightly on the lips. “Obara, so fierce. Nymeria, my sister. Tyene, sweetling. I love you all. The sun of Dorne goes with you.”
“Unbowed, unbent, unbroken,” the Sand Snakes said, together.
A Dance with Dragons, Chapter 38, The Watcher