The put saddles on the dragons, wore armor and fought in tournament type battles. The dragons were not sentient and were only used in the sport.
One day the aliens returned and they, too, had taken prisoners (humans). They bred these humans to be muscle-bound cargo haulers. I believe other things as well.
Book had to be published in 1985 or older if that narrows it down any.