Having just replayed StarCraft 1 up to this betrayal, it was very apparent why Mengsk did it.
In the 3 missions preceding the betrayal Kerrigan she was told to use the psi-emitters to lure the zerg to a smaller confederate outpost. She did it but under heavy protest.
Kerrigan - I'm having doubts about this, Arcturus. I
just don't think anyone deserves to have
the Zerg unleashed on them.
Mengsk - I know you have personal feelings about
this, but you can't let your past cloud your
judgement. Carry out your orders, lieutenant.
later at the end of the mission.
Kerrigan - Psi-Emitter in place. Just promise me we'll
never do anything like this again.
Mengsk - We will do whatever it takes to save
humanity. Our responsibility is too great to
do any less.
In the very next mission we make a breach in Tarsonis' defences, presumibly to send a strike force down to the planet. instead after the Breach Mengsk again deploys psi-emitters, and lures billions of zerg to the planet. This causes a major rift between with Kerrigan/Raynor on one side, and Mengsk on the other.
DUKE
This is Duke. The emitters are secured and on-line.
KERRIGAN
Who authorized the use of Psi-Emitters?
MENGSK
I did, lieutenant.
KERRIGAN
What? The Confederates on Antiga were bad enough, but now you're
going to use the Zerg against an entire planet? This is insane.
RAYNOR
She's right, man. Think this through.
MENGSK
I have thought it through. Believe me. You all have your orders.
Carry them out.
Following Kerrigans damming of Mengsk he sends her on 1 more mission, a mission to stop the Protoss who are unknowingly defending the remaining confederates on Tarsonis.
Mengsk :
If they engage the Zerg, the Confederates may escape.
Commander, send Lieutenant Kerrigan with a strike force to
engage the Protoss. Captain Raynor and General Duke will stay
behind with the command ship.
Raynor :
First you sell out every person on this world to the Zerg, then
you ask us to go up against the Protoss? And you're goin' to
send Kerrigan down there with no backup?
Mengsk :
I have absolute confidence in Kerrigan's ability to hold off the
Protoss.
Raynor :
This is bullshit. Kerrigan, are you reading this?
Kerrigan :
I heard. I'm going down there. Arcturus knows what he's doing. I
can't back out on him now.
Raynor :
Funny. I never thought of you as anyone's martyr.
RAYNOR
Why are you doing this, Kerrigan? Look. I know about your past. I
mean, I've heard the rumors. I know you were a part of those
experiments with the Zerg; that Mengsk came and saved you, but you
don't owe him this! Hell, I've saved you butt plenty of times.
KERRIGAN
Jimmy, drop the knight-in-shining-armor routine. It suits you
sometimes. Just not ... not now. I don't need to be rescued. I know
what I'm doing. The Protoss are coming to destroy the entire
planet, not just the Zerg. I know that because ... well I just know
it. I am a Ghost, remember? Once we've dealt with the Protoss, we
can do something about the Zerg. Arcturus will come around. I know
he will.
RAYNOR
I hope you're right, darlin'. Good huntin'!
This mission is clearly a set up for Kerrigan, with Raynor stating, your going to send her in by herself, with no support, and Mengsk replying, dont worry she'll be fine. Obviously at the conclusion of this mission Mengsk again pulls a fast one and states, uh, cant get to her sorry Jim.
KERRIGAN
This is Kerrigan. We've neutralized the Protoss, but there's a wave
of Zerg advancing on this position. We need immediate evac.
MENGSK
Belay that order. We're moving out.
RAYNOR
What? You're not just gonna leave them?
MENGSK
All ships prepare to move away from Tarsonis on my mark.
KERRIGAN
Uh, boys? How about that evac?
RAYNOR
Damn you, Arcturus! Don't do this.
MENGSK
It's done. Helmsman, signal the fleet, and take us out of orbit.
Now!
KERRIGAN
Commander? Jim? What the hell's going on up there-
So with the books and extra material aside, they make it clear in game that Mengsk betrays Kerrigan because she is quickly becoming a tool he can no longer use, with her arguments and opposition to his plans.