It is pretty much the same as in real-world feudal societies. There is no constitution or any other fixed set of rules. There is a combination of custom, personal charisma, and the strength of one's army and alliances. For example, Robert was a charismatic leader, and he was supported by several very powerful houses, such as the Starks, the Arryns, and the Tullys. To top it off, he was a distant relation of the Targaryens, which satisfied the custom of a hereditary royal line.
On the other hand, Aegon the Conqueror had no hereditary claim to the throne at all. In fact, there was no single throne of Westeros. But Aegon did have dragons.
Compare that to real-world history, and you will see many similarities. Think of William the Conqueror, the War of the Roses in England, or the history of the relations between England and Scotland.