According to Memory Alpha:
Later, at a formal hearing, Kirk drops all charges against Khan and his people, considering it a "waste" to put Khan in a penal colony, and gives him the offer of taming the uninhabited world of Ceti Alpha V an offer which Khan accepts referencing a quote from Milton's Paradise Lost that "it is better to rule in Hell than to serve in Heaven." Marla McGivers is given the option of court martial or accompanying Khan and his people. Khan warns her it will be difficult at first to survive, to find food, and Marla chooses to join Khan and his people.
Stated in the episode:
- Kirk has some appreciation and/or admiration for Khan and his people and doesn't think a penal colony would be justice for them.
- Kirk is empowered to make these decisions.
Conclusions and suppositions that could be drawn:
- Kirk and the senior officers are Starfleet and are empowered to act as they did and make such choices. The idea of turning Khan over to Starfleet doesn't make sense - he is already a prisoner of Starfleet.
- Returning to the nearest starbase might delay or impede the progress of their exploration.
- It may not be safe to allow Khan and his people to remain aboard Enterprise or to be imprisoned by Starfleet pending any other kind of trial.