Renly did not want to remove Joffrey initially
Renly wanted to consolidate his influence in court by allying himself with Eddard Stark and taking role in controlling Joffrey like a puppet much like how Aegon III was controlled by his regency council (Fun fact: His Grace, Aegon III fired his entire regency council on the very day he turned 16).
From AGOT, we see Renly's proposals for Eddard Stark:
“My lord, I have thirty men in my personal guard, and other friends
beside, knights and lords. Give me an hour, and I can put a hundred
swords in your hand.”
“And what should I do with a hundred swords, my lord?”
“Strike! Now, while the castle sleeps.” Renly looked back at Ser Boros
again and dropped his voice to an urgent whisper. “We must get
Joffrey away from his mother and take him in hand. Protector or no,
the man who holds the king holds the kingdom. We should seize
Myrcella and Tommen as well. Once we have her children, Cersei will
not dare oppose us. The council will confirm you as Lord Protector and
make Joffrey your ward.”
Ned regarded him coldly. “Robert is not dead yet. The gods may spare
him. If not, I shall convene the council to hear his final words and
consider the matter of the succession, but I will not dishonor his
last hours on earth by shedding blood in his halls and dragging
frightened children from their beds.”
Lord Renly took a step back, taut as a bowstring. “Every moment you
delay gives Cersei another moment to prepare. By the time Robert
dies, it may be too late... for both of us.”
As evident from this conversation, Renly appeared to believe that Joffrey was in fact the true King and Robert's heir. He did not share any doubts about him, his only concern being what Cersei would do in wake of Robert's death. He appears to be genuinely concerned about his position and possibly life.
So he did not want to remove Joffrey, he wanted to remove Cersei from possible position of Power by taking hold of her children.
Renly's ignorance about Joffrey's parentage
Stannis and Jon Arryn did not trust Renly with their investigations. Unlike Varys and Petyr Baelish, Renly was not very smart when it came to court intrigue so it is unlikely that he knew.
But he was smarter than Eddard Stark and he knew that Cersei would immediately move to fill the court with her supporters and remove everyone who could be a threat to her influence, chief among them would be Robert's family and friends.
He confirmed that he did not know when he offered Eddard to take hold of Joffrey. He later confirmed it in front of Catelyn as well when he said he would not have called his banners if Eddard was regent instead of Cersei.
Then again, when he met Stannis, he again showed that he did not know about the Incest and he did not believe Stannis:
Renly laughed. “You must forgive Lady Catelyn, Stannis. She’s come all
the way down from Riverrun, a long way ahorse. I fear she never saw
your little letter.”
“Joffrey is not my brother’s seed,” Stannis said bluntly. “Nor is
Tommen. They are bastards. The girl as well. All three of them
abominations born of incest.” Would even Cersei be so mad? Catelyn was
speechless.
“Isn’t that a sweet story, my lady?” Renly asked. “I was camped at
Horn Hill when Lord Tarly received his letter, and I must say, it took
my breath away.” He smiled at his brother. “I had never suspected you
were so clever, Stannis. Were it only true, you would indeed be
Robert’s heir.”
“Were it true? Do you name me a liar?”
“Can you prove any word of this fable?”
However, meeting Stannis got him reconsidering his views about Joffrey even though he did not admit it in front of Stannis:
“Do I?” Renly shrugged. “So be it. Stannis was never the most
cherished of brothers, I confess. Do you suppose this tale of his is
true? If Joffrey is the Kingslayer’s get-”
“-your brother is the lawful heir.”
“While he lives,” Renly admitted.
So why proclaim himself King later?
In ACOK, he told Catelyn that the sole reason he proclaimed himself King was Lannister hold on Joffrey:
“On the night of Robert’s death, I offered your husband a hundred
swords and urged him to take Joffrey into his power. Had he listened,
he would be regent today, and there would have been no need for me to
claim the throne.”
“Ned refused you.” She did not have to be told.
“He had sworn to protect Robert’s children,” Renly said. “I lacked the
strength to act alone, so when Lord Eddard turned me away, I had no
choice but to flee. Had I stayed, I knew the queen would see to it
that I did not long outlive my brother.”
Renly had no delusions about his sister-in-law but out of fear for his life and his own ambition, he refused to bow to Stannis or Joffrey and proclaim himself King after forging an alliance with High Garden.
From ACOK:
“It would seem that you are the one who has forgotten Stannis,”
Catelyn said, more sharply than she’d intended.
“His claim, you mean?” Renly laughed. “Let us be blunt, my lady.
Stannis would make an appalling king. Nor is he like to become one.
Men respect Stannis, even fear him, but precious few have ever loved
him.”
“He is still your elder brother. If either of you can be said to have
a right to the Iron Throne, it must be Lord Stannis.”
Renly shrugged. “Tell me, what right did my brother Robert ever have
to the Iron Throne?” He did not wait for an answer. “Oh, there was
talk of the blood ties between Baratheon and Targaryen, of weddings a
hundred years past, of second sons and elder daughters. No one but the
maesters care about any of it. Robert won the throne with his
warhammer.” He swept a hand across the campfires that burned from
horizon to horizon. “Well, there is my claim, as good as Robert’s ever
was.”
Renly believed that now that he had the power, he had no need to bow to Joffrey or Stannis. He could be a King in his own right. That was basis of his eventual claim even though he pretends that it was the Lannisters who forced him to do so.
Renly's High opinion of himself
Renly was always the most non-serious of the Stags. According to their Maester Cressen:
Even as a boy, Renly had loved bright colors and rich fabrics, and he
had loved his games as well. “Look at me!” he would shout as he ran
laughing through the halls of Storm’s End. “Look at me, I’m a dragon,”
or “Look at me, I’m a wizard,” or “Look at me, look at me, I’m the
rain god.”
The bold little boy with wild black hair and laughing eyes was a man
grown now, one-and- twenty, and still he played his games. Look at me,
I’m a king, Cressen thought sadly.
He used to believe that he had what it took to be a better King than both Stannis and Robert and he was determined to prove it:
“Though it’s a fool’s law, wouldn’t you agree? Why the oldest son, and
not the best-fitted? The crown will suit me, as it never suited Robert
and would not suit Stannis. I have it in me to be a great king, strong
yet generous, clever, just, diligent, loyal to my friends and terrible
to my enemies, yet capable of forgiveness, patient-”
“-humble?” Catelyn supplied.
Renly laughed. “You must allow a king some flaws, my lady.”
Why was Ned expecting Renly to help him?
Ned was not initially sure about what to make of the Youngest Baratheon brother. He knew for a certainty that Jon Arryn and Stannis did not trust Renly with whatever they were investigating in brothels of King's Landing.
From AGOT:
Jory went to the wardrobe. “Lord Renly is brother to Lord Stannis as
well as the king.”
“Yet it seems that he was not invited on these rides.” Ned was not
sure what to make of Renly, with all his friendly ways and easy
smiles.
Yet Renly was one of the two witnesses Robert chose while signing the decree for appointment of Eddard Stark as Lord Regent and Protector:
King Robert did not seem to miss her. He bid his brother Renly and
Grand Maester Pycelle to stand in witness as he pressed his seal
into the hot yellow wax that Ned had dripped upon his letter. “Now
give me something for the pain and let me die.”
Eddard never thought that Renly would flee at the hour of need, when fate of Baratheon dynasty would be at stake. He also counted on him to honor his obligation as witness to his brother's last decree.
“When Lord Renly arrives,” Ned said.
Varys gave him a sorrowful look. “I fear Lord Renly has left the
city.”
“Left the city?” Ned had counted on Renly’s support.
“He took his leave through a postern gate an hour before dawn,
accompanied by Ser Loras Tyrell and some fifty retainers,” Varys told
them. “When last seen, they were galloping south in some haste, no
doubt bound for Storm’s End or Highgarden.”
So much for Renly and his hundred swords. Ned did not like the smell
of that, but there was nothing to be done for it.
Afterwards, when Eddard was arrested, he thought Renly shared the blame:
He damned them all: Littlefinger, Janos Slynt and his gold cloaks, the
queen, the Kingslayer, Pycelle and Varys and Ser Barristan, even
Lord Renly, Robert’s own blood, who had run when he was needed most.
So the reason why Eddard counted on Renly was because he was Robert's own brother and would be expected stand to protect throne of his dynasty, even if the person to climb it would be his elder brother.
However, Renly left King's Landing before learning the truth and he never intended to bow to Stannis in any case, despite the miscalculation on behalf of Eddard.
