I had always read that as alluding to gems. Since the ghem, like the haut, tend to be beautiful, long lived and just all around improved humans (at least according to the ghem), I simply read ghem lords as evocative of diamonds and pretty shiny things. Something further supported by their fondness for face painting.
I can find no canon reference to back this up. I did, however, find this:
Proto-IE: *ghem-
Meaning: earth
Hittite: gim(ma)ra- c. 'Feld, Flur' Tischler 573 f
Old Indian: gen. gmá-ḥ (as if from kṣā́ḥ, see *dg'hem-)
Old Greek: khamái̯ `zur Erde hin, auf der Erde'
Russ. meaning: земля
References: WP I 662 f, Buck 16
Comments: Hard to distinguish from the reflexes of the widely spread *dg'hem- #395. All Italic forms (Lat humus, etc.) may in fact belong here.
So, ghem
is actually a proto-indo-european root meaning earth. I have no idea if Ms Bujold's erudition extends to PIE roots (though I would not put it past her) but if so, this lends some support to my theory linking ghem to gem.
Still, in the absence of any canon support this does seem a bit of a stretch, but you never know. I can confirm (since I happen to speak it) that the Greek that is sprinkled around the Vorkosigan books is actual Greek and so, presumably, is the Russian. You've already pointed out the French connection so it does not seem to far-fetched to assume that Bujold is a language geek and might even be aware of the meaning I list above.