This is a very interesting question, and I thought that @suchiuomizu's answer was quite good and upvoted it. But I think it's incomplete. There are two other points which need to be considered:
First, the Valar (and Maiar) were not at all Godlike in their judgment. They made numerous mistakes, generally mistakes of being too trusting -- see the question Why did Manwe not understand the concept of Evil? for a discussion of this. Bottom line: Naivety and bad judgment.
But secondly, oaths were taken very, very seriously in this world. (This is a reflection of the seriousness of oaths in the North in our own world.) Feanor swore an oath to do evil and made it in an unbreakable form:
Then Feanor swore a terrible oath. His seven sons leapt straightway to his side and took the selfsame vow together, and red as blood shone their drawn swords in the glare of the torches. They swore an oath which none shall break, and none should take, by the name even of Iluvatar, calling the Everlasting Dark upon them if they kept it not; and Manwe they named in witness, and Varda, and the hallowed mountain of Taniquetil, vowing to pursue with vengeance and hatred to the ends of the World Vala, Demon, Elf or Man as yet unborn, or any creature, great or small, good or evil, that time should bring forth unto the end of days, whoso should hold or take or keep a Silmaril from their possession.
Thus spoke Maedhros and Maglor and Celegorm, Curufin and Caranthir, Amrod and Amras, princes of the Noldor; and many quailed to hear the dread words. For so sworn, good or evil, an oath may not be broken, and it shall pursue oathkeeper and oathbreaker to the world's end.
Some of Feanor's sons later sought a way out, but failed:
Yet Maglor still held back, saying 'If Manwe and Varda themselves deny the fulfilment of an oath to which we named them in witness, is it not made void?'
And Maedhros answered 'But how shall our voices reach to Iluvatar beyond the Circles of the World? And by Iluvatar we swore in our madness, and called the Everlasting Darkness upon us, if we kept not our word. Who shall release us?'
'If none can release us,' said Maglor, 'then indeed the Everlasting Darkness shall be our lot, whether we keep our oath or break it; but less evil shall we do in the breaking.'
In the end, Maglor wanted to break the oath, and rightly judged that even by the standards of his world that would be the less evil, but he still didn't break it.
Yet he yielded at last to the will of Maedhros, and they took counsel together how they should lay hands on the Silmarils. And they disguised themselves, and came in the night to the camp of Eönwe, and crept into the place where the Silmarils were guarded; and they slew the guards, and laid hands on the jewels.
I think today we look at as pretty silly that someone would swear an oath to God to do evil and think that God would bind them to do that evil. But in the context of the Norse and Middle-Earth, this was solemn and terrible and resolvable only by the direct intervention of Iluvatar.
So on the one hand, the Valar had Feanor (and sons) who were sworn to do evil and had ample proof that they took the oath seriously and would carry it out no matter what forgiveness was forthcoming from the Valar. How could they let them run loose?
On the other hand, they had Sauron and had been running in bad company but who had not made an unbreakable commitment to doing evil. That they could pardon.
In some respects they were right. Sauron was not dedicated to doing evil so much as selfish. Given his "fair hue" it was not unreasonable to think that he could be turned back to good pursuits.