Renly was second in line to the throne, behind Stannis, although Renly may not have known about Joffrey's illegitimacy. However Stannis was not loved by the other Lords or the people, whereas Renly was. For this Renly felt that, encouraged by Loras Tyrell and the Tyrell household, that he would make a better king than Stannis.
Why the oldest son, and not the best-fitted? The crown will suit me, as it never suited Robert and would not suit Stannis.
-Renly to Catelyn Stark
Additionally Robert had made Renly the Lord of Storm's End, which is the seat of House Baratheon, and should have gone to Stannis after Roberted ascended to the throne. This afforded him great support, and enabled him to gather an army of 100,000 from across The Reach and The Stormlands.
Renly also showed that he didn't feel that the normal rules of succession were too important. He made it clear that it was strength and power that won the throne. He once said that the Targaryens didn't have any right to Westeros, they conquered it; Robert didn't have any right to the throne, he took it. Clearly he felt that he would make a better king, had the army to take it, and therefore deserved the crown.
Furthermore, Cersei considered Robert's brothers to be her enemies. Both Renly and Stannis knew this, so Renly would not necessarily have felt safe in a land ruled by Joffrey and Cersei.