1. Because he was still trying to redeem his father:
Luke: Then you know why I have to face him.
Leia: No! Luke, run away, far away! If he can feel your presence, then leave this place! I wish I could go with you.
Luke: No, you don't. You've always been strong.
Leia: But why must you confront him?
Luke: Because there is good in him. I've felt it. He won't turn me over to the Emperor. I can save him. I can turn him back to the good side. I have to try. [kisses Leia on the cheek, then leaves]
2. Because he expected to either save his father or die trying:
Palpatine: Ah, yes, a Jedi's weapon, much like your father's. By now you must know that your father can never be turned from the dark side. So will it be with you.
Luke: You're wrong. Soon I'll be dead, and you with me.
3. Because he couldn't hope to kill the Emperor without doing so in hate, which would turn Luke to the Dark Side:
Emperor Palpatine: [laughing] Good! Your hate has made you powerful. Now, fulfill your destiny and take your father's place at my side!
Palpatine: I can feel your anger. I am defenseless. Take your weapon. Strike me down with all of your hatred and your journey towards the dark side will be complete!
Palpatine: You want this, don't you? The hate is swelling in you now. Take your Jedi weapon. Use it. I am unarmed. Strike me down with it! Give into your anger! With each passing moment, you make yourself more my servant.
Palpatine: It is unavoidable. It is your destiny. You, like your father, are now...mine.
Palpatine: Good! Use your aggressive feelings, boy. Let the hate flow through you!
4. Because he still believed that the Death Star, and Palpatine with it, would soon be destroyed either way:
[See quote under #2, above]
5. Because Palpatine clearly wanted him to try, and it is best to not do what Palpatine wants you to do:
[See quotes under #3, above]
6. Because he didn't know that Force Lightning existed, so he probably assumed that Palpatine was unarmed and defenseless; thus, he would be killing an unarmed man:
The young Jedi was all at once confounded and in agony—he’d never heard of such a power, such a corruption of the Force, let alone experienced it.
- Return of the Jedi - The Official Novelization
7. Because it
probably definitelywouldn't have worked anyway:
Here's what happened when four highly trained Jedi Knights, who had been studying under a large number of powerful Jedi Masters since they were small children, took on the Emperor:
It took Palpatine TEN SECONDS to kill THREE of the best Jedi in the galaxy. They had decades of training under their belts. Luke, on the other hand, spent A WEEK with Obi Wan, and maybe another month with Yoda. He wouldn't have stood a chance of even hurting the Emperor, let alone killing him. He just wasn't good enough.
Even Yoda couldn't beat Palpatine in a lightsaber fight:
Compare the lightsaber techniques of Mace Windu, Yoda, and Emperor Palpatine to Luke's technique in his final duel with Vader:
This is a sledgehammer approach. He's just flailing away as hard as he can, hoping that something lands. The others make lightsaber combat look like an art; Luke makes it look like he's trying to whack a piñata. He doesn't have what it takes to beat a true pro. Which is why this happened:
Palpatine raised his spidery arms toward Luke: blinding white bolts of energy coruscated from his fingers, shot across the room like sorcerous lightning, and tore through the boy’s insides, looking for ground. The young Jedi was all at once confounded and in agony—he’d never heard of such a power, such a corruption of the Force, let alone experienced it. But if it was Force-generated, it could be Force-repelled. Luke raised his arms to deflect the bolts. Initially, he was successful—the lightning rebounded from his touch, harmlessly into the walls. Soon, though, the shocks came with such speed and power, they coursed over and into him, and he could only shrink before them, convulsed with pain, his knees buckling, his powers at ebb.
This would have happened regardless of whether or not Luke was willing to fight the Emperor. And if he had, it is likely (in my opinion) that Vader would have been less inclined to intervene on Luke's behalf. Vader stepped up to the plate because his son was being tortured to death while unarmed and incapable of defending himself, not because he picked a fight he couldn't win.
Remember, Luke's primary objective was not to kill the Emperor - he figured that the Emperor would die when the Death Star was destroyed anyway. His primary objective was to save his father, and he did so the only way he thought possible:
Luke: Never. I'll never turn to the Dark Side. You have failed, Your Highness. I am a Jedi, like my father before me.
Palpatine: [angrily] So be it... Jedi!
Luke refuses to give in to Palpatine's demands and disarms himself.]
Palpatine: [raising his hands toward Luke] If you will not be turned, you will be destroyed! [Fires lightning bolts at Luke, causing him to fall to the floor in agony. Darth Vader gets up and stands next to Palpatine, watching.] Young fool. Only now, at the end, do you understand. [shoots another round of lightning] Your feeble skills are no match for the power of the Dark Side! You have paid the price for your lack of vision! [continues shooting lightning]
Luke: [writhing in agony] Father, please! Help me!
Palpatine: Now, young Skywalker... you will die. [intensifies lightning blasts]
Darth Vader/Anakin Skywalker: [looks back and forth at Luke and Palpatine. He grabs Palpatine from behind and throws him down into the reactor shaft]
[A redeemed Anakin Skywalker is dying in Luke's arms]
Anakin Skywalker: Luke, help me take this mask off.
Luke: But you'll die!
Anakin: Nothing can stop that now. Just for once, let me look on you with my own eyes. [Luke carefully removes Vader's mask to reveal Anakin's disfigured face underneath.] Now...go, my son. Leave me.
Luke: No, you're coming with me. I won't leave you here. I've got to save you!
Anakin: [smiles] You already have, Luke. You were right. You were right about me... Tell your sister... you were right... [dies]
The point is this: Luke couldn't have beaten the Emperor in any other way. His only hope was to get his father to help him. Which is precisely what he did.
As for your subquestion:
Why would he waste this perfect opportunity to safely kill the Emperor the way Yoda and Ben would have wanted?
I've already made it clear that, in my opinion, Yoda never suggested that Luke should kill Vader, and Obi Wan may have made a mistake in saying that Luke had to either kill Vader or fail. Even if they both really did want Luke to kill Vader, they are clearly very happy to have been proven wrong:
Remember: Even great Jedi Masters are sometimes wrong.