The Governor's actions are actually all driven by fairly rational processes, and chalking everything he does up to "he's just insane" does a huge disservice to a very complicated character.
The reason that the Governor has no choice (in his mind) but to kill Martinez is that Martinez is the last person alive who knows his past. Martinez was there for all of the worst things the Governor has ever done, up to and including his butchering of all those who followed him into battle against the prison group. It's made very clear in the preceding events that the Governor is trying desperately to keep all of that hidden. It's pretty much explicit in the scene where his new group has just found a pack of beer and is sitting down, chatting, and he has to quickly cut Martinez off from exposing him.
Mitch: I can never tell if he's winkin' or blinkin'. But you know how to regulate, don't you, Bry?
[silence]
Mitch: He was always like this, Martinez?
Martinez: Oh, yeah. Ice in the veins. You should've seen him back in the day--
The Governor: What about you fellas, huh? What'd you do before it all changed?
Obviously if any of the information about his previous life came out, it would jeopardize not only his position within the new group but, perhaps more importantly, his position within his new "family".
Now let's examine the murder dialogue and sequence of events.
Martinez: Your family, they brought you back. You're lucky. Couldn't... I couldn't do that again. Couldn't risk it. Couldn't sleep at night, knowing I was gonna lose 'em.
The Governor: I'm not gonna lose 'em.
Martinez: Yeah.
The Governor: What, you, uh... you don't think you can keep this place safe?
Martinez: I'll try. Hopefully we'll be prepared for whatever comes.
It's as Martinez says this line that the Governor looks up and has clearly decided to kill him. You can see his face change right there. And upon the next line, the golf club comes down on his skull.
Martinez: Now that you're here, maybe we could share the crown a little. [hits golf ball] Ah, Jesus... I shoulda taken some golf lessons. Before the--
The Governor already knew before this that Martinez was a liability and would have to be dealt with in one way or another. He knew he would probably have to kill the man. But when Martinez told him that he was certainly going to lose his new family, partially because he had no confidence in his ability to keep his people secure, it cemented it in his mind. Martinez had to be removed, both because of his knowledge and because of his inability to lead.
So, what do his cries of "I don't want it!" mean?
Obviously that's open to literary interpretation and opinion. However I believe that he is being genuine in that moment. He doesn't want the power. He doesn't want to lead again. He would have been perfectly happy just joining some group and being with his family and doing his part, raising his new daughter, being with his new wife. That's why he burned up the photo of his old family, after all. They were irrelevant now. These new people were his life.
But the group he ran into just happened to be one that contained the only person living who could expose him for the vile man that he was, or that he had once been. It had the only person who could with just a few sentences take away the amazing second chance that he had been given. And in a stroke of even worse luck, it was that man who happened to be in charge of that group.
He didn't want to lead. He didn't want to kill anyone else. But he had no choice. He couldn't allow anyone to know what he had done. And he couldn't allow his family to come to harm due to incompetence.
He didn't want it.
It's also very clear that he doesn't want the power at all, because even after Martinez is dead he doesn't try to seize it. He lets Pete be the leader. And when things start to go badly within the group and the internal power struggle happens, he doesn't take part in it. He tells his family to get into the camper and to pack their things. They try to flee. He genuinely doesn't want to kill anyone else.
But when their camper is blocked by a giant puddle full of Walkers, he has to turn back around and go back to that broken group. And it's at the point that he believes he has no choice. If he can't leave the group, and if the group is lead by morons who are going to get the people he cares about killed... well, he has no choice but to lead it.
