This is the "Faded Sun Trilogy" by C. J. Cherryh
The castes you're referring to are the Kel, the Sen and the Kath
Per wikipedia;
The mri are divided into a rigid structure of three castes: kel, sen,
and kath. These are functional rather than hereditary castes, but each
has a clearly defined role in their society, and the mri regard any
change in those traditional functions as completely unacceptable.
Kel are warriors, and dress in black. Because they interact with
outsiders, they veil their faces to preserve the privacy of the
People. Furthermore, they are forbidden to read or write, lest they
inadvertently betray the ancient secrets of the People. They are
masters of both the traditional edged weapons of the People and the
modern weapons they use as mercenaries. They play shon’ai, the
traditional passing game of the People, with knives.
Sen are scholars, and dress in gold, except for the she’pan or
matriarch of the tribe, who dresses in white. They keep the sacred
writings of their people, and are celibate. Because they do not
interact with outsiders, they do not veil their faces. They play
shon’ai with wands.
Kath are the children and those women who do not join either the kel
or the sen, and dress in blue. No man past puberty may remain in kath,
and those unsuited for kel or sen die. The kath are the lowest caste
of the mri, responsible for the upkeep of the tribe's dwellings. Kath
play shon’ai with a pair of rounded stones.