Theoretically the entire Culture sets the policy of Contact (and its dirty-tricks division, Special Circumstances) but the realpolitik answer is that they (in quorum) set their own goals and aims.
Politics in the Culture consists of referenda on issues whenever they are raised; generally, anyone may propose a ballot on any issue at any time; all citizens have one vote. Where issues concern some sub-division or part of a total habitat, all those - human and machine - who may reasonably claim to be affected by the outcome of a poll may cast a vote. Opinions are expressed and positions on issues outlined mostly via the information network (freely available, naturally), and it is here that an individual may exercise the most personal influence, given that the decisions reached as a result of those votes are usually implemented and monitored through a Hub or other supervisory machine, with humans acting (usually on a rota basis) more as liaison officers than in any sort of decision-making executive capacity; one of the few rules the Culture adheres to with any exactitude at all is that a person's access to power should be in inverse proportion to their desire for it. The sad fact for the aspiring politico in the Culture is that the levers of power are extremely widely distributed, and very short (see entry on megalomaniacs, above). The intellectual-structural cohesion of a starship of course limits the sort of viable votes possible on such vessels, though as a rule even the most arrogant craft at least pretend to listen when their guests suggest - say - making a detour to watch a supernova, or increasing the area of parkland on-board.
A FEW NOTES ON THE CULTURE by Iain M Banks
Note that most Culture citizens show little interest in what's happening in the wider galaxy, save for what impacts their own insulated lives.
So, yes, these people. They share collective responsibility for the
actions of their Minds, including the Minds of Contact and Special
Circumstances. That’s the way they’ve set it up, that’s the way they
want it to be. There are no ignorants here, Quil, no exploited, no
Invisibles or trodden-upon working class condemned forever to do the
bidding of their masters. They are all masters, every one. They can
all have a say on everything. So by their own precious rules, yes, it
was these people who let what happened to Chel happen, even if few
actually knew anything about the details at the time.
Look to Windward
As to what (and how) they justify their actions to the wider public, the answer is that they have cold, hard statistics to show that their involvement produces more good than harm.
“We are,” Anaplian assured him. “Constantly. Still, we can prove that it works. The interfering and the dirty-tricking; it works. Salvation is in statistics.”
“I wondered when we’d get to that,” the man said, smiling sourly and nodding. “The unquestioned catechism of Contact, of SC. That old nonsense, that irrelevance.”
“Is not nonsense. Nor . . . It is truth.”
The man got down from his bar stool. He was shaking his head. This made his wild fawn hair go in all directions, floatily. Most distracting. “There’s just nothing we can do,” he said sadly, or maybe angrily, “is there? Nothing that’ll change you. You’ll just keep doing all that shit until it collapses down around you, around us, or until enough of everybody sees the real truth, not fucking statistics. Till then, there’s just nothing we can do.”