An Earth man is picked (or sent by his boss/superior) on a one way trip to a distant empire (their emperor had some mad daydream visions, which everyone chose to ignore, for fear of being seen as admitting the emperor's sickness).
The dominant class's main trait was their height and enhanced capabilities (velocity, strength, intelligence).
The nobleman/tall aliens duel often (using some kind of rings??) The Earthling is given a military honorary post (if I'm not mixing my stories). In time the Earthling climbs the nobleman value ladder and uncovers a threat to the emperor and is rewarded with his return back to Earth.
He returns with a female nobleman love interest (which I believe it was less tall than the other members of the ruling class being perceived because of that as inferior by them).