First of all, it is clear that Khan deliberately initiates the overload, as seen at the beginning of this clip:
Khan points to the alarm right above the console that he was interacting with just before Kirk entered the room.
Earlier in the episode, Khan had used his genetically-enhanced intellect to memorize and correctly interpret all of the Starfleet technical manuals regarding the Enterprise. This is how he was able to initiate the overload in Engineering. It is impossible to say whether he planned this from the moment he escaped the briefing room or not, but it is likely that he decided shortly after reading the manuals that this course of action could be useful as a contingency.
As for his motivations, my interpretation of the scene in Engineering is that he is not "giving up". Rather, he does not believe that Kirk — who has been easily disarmed — will be able to physically overtake him:
(Kirk dashes into Engineering and straight into Khan, who easily disarms him then crushes the phaser with his bare hands. Then an alarm beeps.)
KHAN: If I understood your manuals, that's an overload in progress. Your ship flares up like an exploding sun within minutes.
(The two fight. Khan can throw Kirk around with little effort.)
KHAN: I have five times your strength. You're no match for me.
Having measured up the kind of man that Kirk is, Khan would believe that even if Kirk were willing to sacrifice himself to stop Khan, he would not be willing to sacrifice his crew of roughly 400 people. Contrast this to Khan's relationship to his followers, as described to Chekov and Terrell in The Wrath of Khan:
KHAN: These people have sworn to live and die at my command two hundred years before you were born!
Khan could be working under the following assumption: since Kirk will not be able to overpower him and get to the controls, Kirk will surrender and can be made to give an order to vent the neural gas (which has knocked out Khan's followers) in exchange for disabling the overload, thus sparing the crew.
What Khan doesn't anticipate is Kirk's intimate knowledge of Engineering: Kirk decides to use a tool that was partly hidden in a control panel to beat Khan unconscious.
By the way, if Kirk hadn't entered the room, one can imagine that Khan would have used the ship's intercom to announce that the Enterprise was about to be destroyed unless control of the ship were returned to him and his followers resuscitated. In any case, Khan is a quick study, and we ought to believe that he was forming some kind of plan to regain control of the ship (if not this plan, then something else).