Baratheons didn't legally inherit the throne. They usurped the crown and killed the lawful King along with his heirs (Two of whom fled into exile). While it is true that Robert's claim to the throne came from his descent from Targaryens (His grandmother was a daughter of King Aegon V and aunt to the Mad King Aerys), but that didn't make his rebellion lawful or him the lawful heir. Robert was in the line of agnatic primogeniture succession for King Aerys II which would have looked like:
1.1 Aegon Targaryen
Had the Targaryen Princelings died of some accident/natural causes or joined some order whose vows ended their claims and rights to hold lands, Then Robert would have been the rightful heir to Aerys II and the rightful King for the whole realm. But we know, that didn't happen, a rebellion happened.
With King Aerys II, Prince Rhaegar, Prince Aegon, Princess Rhaenys, Queen Rhaella dead and Prince Viserys, Princess Daenerys fled, there were no Targaryens left to lead the loyalists. King's Landing was in Robert's hands and six of the Eight great Houses firmly behind Rebels (Only Tyrells and Martells remained loyal to the Bitter end), the Loyalists had no choice but to bend the knee. While they did bend the knee, Robert feared that they were only biding their time until Viserys could land with an army. It wasn't just the lords, we see plenty of times in the books how common people and knights openly show disdain for Baratheons and yearn for return of the Dragons, calling them the only rightful Kings.
Just because Aerys II was burning people for imagined or real crimes doesn't mean it was legal to usurp the crown. He's not the first tyrant who ever took the throne. King Maegor was the first, he was allegedly killed by his own throne (In reality, it was probably his Queens or the Kingsguard). King Aegon II was poisoned ostensibly by his own council members when he refused to accept their counsel to make peace with the Blacks. As Lord Cregan Stark commented, on Aegon II's murder:
To kill a cruel and unjust king in lawful battle was one thing. But foul murder, and the use of poison, was a betrayal against the very gods who had anointed him.
Had Rebels only killed Aerys and proclaimed a new Dragon King, it would be legal. The Lannister bannermen who were first to enter the Red Keep weren't sure if they were going to have a new Targaryen King or not. As it happened however, Tywin Lannister answered their questions with corpses of Rhaegar's children.
Of course for Baratheon loyalists, the throne belonged to the Stormlords now. But for Targaryen loyalists, The Stormlanders had usurped the throne which they had no right to. For them, only Targaryens are the rightful Kings. Legality and rights are subjective to whoever you are talking to. Of course, the matter is less complicated since Baratheons are now extinct in male line, Nobody fights for dead men. Targaryens are however very much alive and as far as Loyalists are concerned, they are the only true heirs to the Iron Throne.