There was a book I read in the 1980's, a Science Fiction/Fantasy paperback where a phoenix bird lived in a garden, ate jewels like rubies and emeralds. A sorcerer or wizard was one of the characters and there was some kind of garden where the 'lost color' bloomed in a flower. I loved that story but cannot remember the title or author.
There was some kind of magical quest, a young man and woman in a potential romantic relationship. The book was less than 300 pages, the cover may have been tan or beige with a painting/drawing of a castle or map which included a picture of a castle, garden. I know I bought it at Browseabout Books in Rehoboth Beach, Delaware. It was not a series or a trilogy, just a one-off.
There was also wry, dry humor in it, making me think the author was British.