"Pity? It was pity that stayed Bilbo's hand. Many that live deserve death, and some that die deserve life. Can you give it to them, Frodo? Do not be too eager to deal out death in judgment. Even the very wise can not see all ends. My heart tells me that Gollum has some part to play yet, for good or ill…
Sauron was a Maiar of a "far higher order"‡ than Gandalf himself, and was not immediately turned to Melkor; he was, in fact, a great apprentice to Aulë the Smith and grew to be a great craftsman in his own right (hence his ability to craft the Rings of Power.)
We know that Olórin (Gandalf) knew Sauron - at least well enough to fear him‡‡, which is what caused the Valar to appoint Curumo (Saruman) as head of the Istari. Given what @wad-cheber already shared about Olórin's time with Nienna, and his conversation with Frodo about the power of pity (quoted above), it's not unusual to imagine that Olórin wept at the loss of Sauron - while he lived, and was a threat, Olórin dared not admit that pity into his heart for fear of not being able to deliver a felling blow should it be required. However, as Sauron was destroyed, he could finally allow himself to weep for the great loss.
‡In "The Letters of J. R. R. Tolkien" (Tolkien, J. R. R., Carpenter, H., & Tolkien, C. (1981). The letters of J.R.R. Tolkien. Boston: Houghton Mifflin.), near the end of the entry "Notes on W. H. Auden’s review of The Return of the King" (Letter no. 183), we find the following phrase:
In my story Sauron represents as near an approach to the wholly evil will as is possible. He had gone the way of all tyrants: beginning well, at least on the level that while desiring to order all things according to his own wisdom he still at first considered the (economic) well-being of other inhabitants of the Earth. But he went further than human tyrants in pride and the lust for domination, being in origin an immortal (angelic) spiritfn54.
Following the footnote, we find the following sentence:
fn54: Of the same kind as Gandalf and Saruman, but of a far higher order.
Also, in "Unfinished tales", (Tolkien, J. R. R., & Tolkien, C. (1980). Unfinished tales of Númenor and Middle-earth. Boston: Houghton Mifflin.), we find this phrase:
And Curunír, Saruman the White, fell from his high errand, and becoming proud and impatient and enamoured of power sought to have his own will by force, and to oust Sauron; but he was ensnared by that dark spirit, mightier than he.
‡‡ In "Unfinished tales", during the council of the Valar, as they're deciding which Maia should be sent to Middle-Earth "and clothe themselves in flesh so as to treat on equality and win the trust of Elves and Men…," Manwë specificallys commands Olórin to go:
Olórin, who was clad in grey, and having just entered from a journey had seated himself at the edge of the council, asked what Manwë would have of him. Manwë replied that he wished Olórin to go as the third messenger to Middle-earth…But Olórin declared that he was too weak for such a task, and that he feared Sauron.