Short answer: they were killed through war, disease and horrible disasters.
The 300 years of Targaryen rule has been extremely bloody. And while the people of Westeros stood no chance against the dragon riders, the Targaryens were not immune against themselves. Several incidents served to whittle down the Targaryen clan (which was never very large in the first place).
The Dance of the Dragons: The first Targaryen civil war. When King Viserys I died, a successsion crisis erupted between his heir apparent Rhaenyra Targaryen and her half brother Aegon (the eldest male child). The Targaryens (and Westeros) were split in half and a bloody war ensued that saw both sides utilizing dragons. In the end Aegon II was crowned king, but not before both sides suffered great losses in both Targaryens and dragons. It was this war and it's immediate aftermath that lead to the decline of the Targaryen dragons You can read more about that war in the prequel novella The Princess and the Queen.
The Blackfyre Rebellion: The second Targaryen civil war. Just before King Aegon IV died he legitimized a number of his bastards, chief among them Daemon Blackfyre. When Aegon died, many noblemen could not stomach his bookish heir Daeron II and flocked around the classically martial Daemon who claimed the Iron Throne for himself. The rebellion was quelled, but again not without serious loss of life. Heirs of Blackfyre would continue to plague Westeros, later even managing to kill King Maekar I.
The Great Spring Sickness: A plague that ravished Westeros leaving tens of thousands of dead in its wake. Amongst the victims were King Daeron II and several Targaryens including several of the king's grandsons.
The Tragedy of Summerhall: A mysterious event that killed several Targaryens, including King Aegon V and his heir. Fandom speculated that it was an attempt to revive dragons that went horribly wrong.
Robert's Rebellion: 'Nuff said.