Oh, I know this one. It’s Circle of Magic, by Debra Doyle. Primarily because I've read the book and remember it matching what OP describes, but I did a little digging and found this on the author's website:
Just before noon, Randal found Madoc in the tower. The wizard was reading a small, leather-bound book.
"What is it, lad?" asked Madoc, not looking up.
"I—I want to be a wizard like you," Randal told him.
"How can you want to be a wizard, boy? You haven't go the foggiest idea of what it's all about." Madoc rose and stood glaring down at Randal. "You'll spend most of your life with just enough power to get you into trouble. You'll be hungry more often than you're fed. You'll spend more time in danger on the road than safe under a roof. And maybe you'll survive it all and live to be old and white-bearded and wise—but if you do, most of your friends will have died a long time before. Go back downstairs to your uncle, lad, and one day you'll make a fine knight. Wizardry is no life for you."
Randal went, but he felt restless and uneasy. Even if wizardry was as hard as Madoc said, it was still the only thing he wanted.
However, Madoc does not wear gray robes. He usually wears saffron robes and a gray kilt.
The newcomer wasn't much to look at: a man about forty years old with a short dark beard, carrying a walking staff taller than he was. He wore a loose shirt of faded yellow linen and a rough kilt of gray wool, belted around his waist and folded up over one shoulder. He's a long way from home, thought Randal. Only the half-civilized tribesmen of the north country dressed like that.
The students wear black robes.
All of the young people wore loose black
robes over their regular garments; Randal wondered if the
robes had some sort of significance.
The Masters at the school wear nicer black robes with more colorful tunics.
The other four wizards wore
heavy, velvet-trimmed robes of rich black cloth, with deep
hoods thrown back to reveal linings of vivid satin. The
northerner still wore his familiar gray kilt and saffron tunic,
but a similar robe hung over the tall chair back behind him.