I'm looking for a story in which the hero's enemy (or rival) takes most of the heat protectant salve and spreads it on his body like a golden skin, so that the enemy can get through the fiery heat undamaged. The hero then takes the pitiful remainder and applies it very thinly, but enough to get through also. It's an ordeal for the hero to get through the heat, and the fire burns away the salve and hurts, but the hero makes it through. The enemy, though, has applied it too thickly and becomes petrified as the unburned salve that he's wearing solidifies into a hard shell once he's out of the fire. It could be a short story or a scene in a larger novel; it is not the climactic twist, though, the hero continues onwards.
Master of the Five Magics, by Lyndon Hardy.
Originally published in 1980, with this cover:
The scene you describe is the climax of the book's second section, when the main character, Alodar, is learning Alchemy.
He and his master have gone into debt to a local businessman in order to develop the heat-protective salve. There's a mine nearby with jewels that are trivial to recover - except that the magma/lava in the mountain causes the temperature to be so high that people can't stand to be there.
When Alodar and his master have created some of the salve, the businessman has a goon steal it (if they didn't go get the gems the next morning, they'd go into indentured servitude to the businessman; given the nature of the work to be done, they'd be unlikely to make it through that period).
Alodar realizes there's a little bit of the salve left on the floor. He finds the drops of it (apparent, it acts much like mercury, cohering into little balls of fluid), and applies them to his body.
The goon uses the salve that had been stolen, lathering it all on (as opposed to the very thin layer that's all Alodar can smear on). Alodar and the goon go into the mine, and what the OP described happens: The heat evaporates the thin layer of salve on Alodar; the thick layer on the goon results in his death.
The gems brought back cover Alodar's (and his master's) debt; in addition, he finds an artifact that leads him to study a third branch of magic (and the book goes on to that section).
This is Master of the Five Magics by Lyndon Hardy.
The book (and its sequels) tells the story of Alodar, as he journeys through the learning of the Five Magics (Thaumaturgy, Alchemy, Magic, Sorcery, and Wizardry) in a quest that starts off trying to win the hand of the queen, but becomes a fight against a demon invasion where only the Archmage (or Master of the Five Magics) can prevail.
The bit you remember is where one of the protagonists sends his henchman (Rendrac) into a pit of fires to gain a treasure of gems to aid the queen and bolster her treasury.
Alodar is at this time an apprentice to an Alchemist and creates a potion that can be used to deflect the heat of the fires, but must be used sparingly.
“You did as well as you were able, Saxton,” Alodar said as he looked about the room. Anger began to boil. “I will pursue and give Rendrac his due. We shall recover the ointment yet and your treasure as well.”
“It is too late for that, my lad,” Saxton said, beginning to breathe with difficulty. “I have studied the inner organs of animals enough to guess what has happened to me. I am not to partake of any of the jewels of the mountains.”
He stopped, and a deep sigh rattled through his lungs. “But then neither will Basil have his way,” he continued. “Two successes with a caloric shield! It is enough for any alchemist.”
“Sweetbalm, or perhaps thaumaturgy,” Alodar said. “We have quested, Saxton, you and I. Do not falter when the goal is in sight.”
“All the balms were destroyed in the mess.” Saxton waved one arm in a feeble arc over his head. “Think no more of me. Flee instead while you can. Basil will receive enough from Rendrac’s trip into the heart of the mountains to care little for the service of a novice.”
“Rendrac braves the heat?” Alodar asked.
“Yes, he anointed himself with the full contents of one flask as I looked on helplessly,” Saxton replied. “When he was done, he resembled less a man than a silver demon, the coating did shine so. And the second batch he crushed underfoot and rubbed its precious salve into the muck he already had made. The other two flasks on the roof were destroyed as well, I fear, when he tossed all the gear to the earth in his rage to find the ointment.”
Alodar applies what is left of the Caloric Shield ointment and ventures in returning with a small chest of gems as well. Rendarc attempts to take the chest of him and they battle
In disbelief, Alodar looked to Rendrac’s face and then back to the weapon still in midswing. As Alodar watched, it slowed to a crawl and then stopped motionless. Almost simultaneously, the big man uttered a weak yelp, and his free hand slowly rose with a spasm of effort from his waist to a mouth held rigidly open under eyes filled with fear. For a second, nothing happened and then, like a silver statue, Rendrac toppled to the ground with a loud clang. Alodar moved to the prostrate form, its limbs still in the rigid position they had held when erect. He reached out and touched the hand that held the blade and felt a deep coldness, rock-hard and smooth, Alodar struck down with his own sword, pommel first, onto an outstretched rigid arm. The now inert form rang from the contact.
“The ointment,” Alodar murmured. “It was meant to be used sparingly and burnt off. Rendrac was too greedy and applied too much. And now it has degraded with age and entrapped him.”
The subsequent books are Secret of the Sixth Magic and Riddle of the Seven Realms.