I think this is In Case of Fire by Randall Garrett.
The following all tally pretty closely:
- Humans were at war with space aliens. These aliens decide to start peace negotiations.
The Karna, slowly being beaten back on every front, were suing for peace. They wanted an armistice conference—immediately.
- Hoping to get beneficial terms they pick a distant unimportant outpost, knowing that the official diplomats would take a long time to arrive.
Already, they had taken the offensive in the matter of the peace talks. They had sent a full delegation to Saarkkad V, the next planet out from the Saarkkad sun, a chilly world inhabited only by low-intelligence animals. The Karna considered this to be fully neutral territory, and Earth couldn't argue the point very well. In addition, they demanded that the conference begin in three days, Terrestrial time.
The trouble was that interstellar communication beams travel a devil of a lot faster than ships. It would take more than a week for the Earth government to get a vessel to Saarkkad V. Earth had been caught unprepared for an armistice.
- It falls upon the staff of outcasts and screw-ups to negotiate.
So Earth had to get a delegation to meet with the Karna representatives within the three-day limit or lose what might be a vital point in the negotiations.
And that was where Bertrand Malloy came in.
He had been appointed Minister and Plenipotentiary Extraordinary to the Earth-Karn peace conference.
- The base commander uses the psychological traits of his staff to assemble a negotiating team.
It's actually more than just making the best of what you have:
Malloy didn't like to stop at merely thwarting mental quirks; he
liked to find places where they were useful.
A few differences are that the protagonist is actually the ambassador to the Saarkkadians (a different set of aliens), rather than the commander of an outpost, and the peace talks take place on another planet in the same solar system. Furthermore, the staff are not quite outcasts and screw-ups. It's just that this particular planet isn't very important and all the best people get assigned elsewhere.
But Malloy didn't have top-grade men. They couldn't be spared from
work that required their total capacity. It's inefficient to waste a
man on a job that he can do without half trying where there are more
important jobs that will tax his full output.
So Malloy was stuck with the culls. Not the worst ones, of course;
there were places in the galaxy that were less important than Saarkkad
to the war effort. Malloy knew that, no matter what was wrong with a
man, as long as he had the mental ability to dress himself and get
himself to work, useful work could be found for him.
However, everyone mentioned in the story has at least one psychological disorder. In particular, the negotiating team consists of one man who is paranoid and another who can't make important decisions if there is more than one option. Together they form the best possible team against aliens who are known to be very cunning negotiators. The ambassador explains to the Secretary of State that is is because
"... those men, as a team, are unbeatable because, in this situation, they're psychologically incapable of losing."
NB The ambassador wasn't actually "normal" - he had a couple of mental quirks of his own but it turns out these are actually perfect for an ambassador in the social structure of the Saarkkadians, suggesting that his own position is another example of finding the right role for his mental make-up.