After Ellaria Sand stabbed Prince Doran Martell, she becomes the de facto ruler of Dorne.
If she is not of the royal bloodline, why is she allowed to rule?
Wouldn't one of Doran's relatives become the next ruler?
Your question asks about both the show and the books (judging from your tags). In the books, however, Doran Martell is for the time being alive and well. He also has several children who are his direct heirs: Arianne, who is being groomed to succeed him (in Dorne, women have succession rights), Quentyn, and Trystane Martell. Ellaria Sand also apparently has no plans to kill Doran in the books, because Doran is not against taking action against the Lannisters (his inaction in taking revenge against them fueling her murder of him in the show).
In the show, Doran Martell only has one heir: Trystane Martell. Arianne and Quentyn do not appear. You're completely right that Trystane would succeed his father upon his death. However...
Ellaria's daughters, the Sand Snakes, kill Trystane as well, wiping out House Martell.
Ellaria Sand then usurps the power in Dorne. She has no legal claim to the throne, but House Martell's army was apparently as displeased as Ellaria (and House Tyrell) was with Doran's inaction against House Lannister, as they just stand by and watch as Doran is murdered.
You ask how she becomes the "de facto" ruler of Dorne, but you also ask about succession and heirs. If Ellaria had any legal claim to the throne of Dorne, she would be the de jure ruler of Dorne. That is not the case. Becoming de facto ruler of anything does not require a legal claim, as we've seen several times in the series. Remember: "Power resides where men believe it resides. No more and no less." With the army and House Tyrell on her side, and no obvious legal claimants to the throne, Ellaria becomes the new ruler.
See also this question for more thoughts on how they pulled it off in practice.