Stannis' biggest concern at that time was getting an army big enough to challenge all the claimants. He felt the right to only one army, the Stormlander one which was sworn to Renly.
Stannis first tried to get Davos Seaworth to gather the Stormlander Lords around him by diplomacy. That didn't work. More courteous Lords met Davos but refused to help him while the straightforward ones refused to see Davos at all.
Stannis had always felt that Robert had cheated him of his birthright by giving Storm's End to Renly instead of him. As Steffon Baratheon's second son, Stannis felt that after Robert's coronation, Stormlands were his by birthright along with all her armies and incomes.
He could have used the shadow to kill Robb Stark, Joffrey Baratheon or even Balon Greyjoy but what purpose would that serve? Northmen wouldn't bow to him, Lannisters would crown Tommen and Greyjoys would choose one of their own from Balon's family. Killing Renly however gave him what he needed; the Stormlander Army. After Renly's death, Stannis was the last Baratheon. Where else could the Stormlander lords go? He could of course sire another Shadow and get rid of them all but Melisandre said his fires were burning low (Of course that happened after the Battle of Blackwater bay and Stannis didn't know about that - yet he didn't consider that course and used it only when he deemed it absolutely crucial for victory).
And even more importantly as far as Stannis was concerned, at Storm's End was the proof he needed to prove his allegations against Cersei and Jaime; Robert's bastard Edric Storm. He couldn't get his hands on the boy without dealing with Renly first. Submitting to Renly in exchange for the boy would have been out of question, Renly had to die or bend the knee.
That's why Stannis chose Renly but not before giving him a final chance to submit (To be fair, Renly offered Storm's End to Stannis if he swore fealty but Stannis thought Storm's End was his anyways and not Renly's to give). He even offered to make him his heir until a son was born to him. Renly refused and sealed his fate. It was a mixture of your guess #2 and guess #3.