Name the novel, 197x. Setting is a world of human(oid)s most of whom have the psi ability to enslave weaker others. (Maybe the word slave and its derivatives are not used.) Masters and slaves form a tree rather than a dichotomy; everyone delegates as much as they can.
The protagonist is an outworlder on a mission, maybe simply doing a survey. He wears the green robe of a native subculture who lack the power (or maybe they renounce it?) and are immune to it – by law or custom, I think, rather than by nature.
His host or primary contact is Jael Forty of Lant, the number being how many slaves he can control; it can fluctuate.