I don't believe this is explored in-series at all.
That said, the most obvious speculative answer is politics.
The quarians effectively created their own monster and enemy. They bungled the geth so badly they lost their own homeworld and were reduced to space pilgrims. They had nothing to offer the Council in return. They asked for intervention; effectively, for the Council races to directly engage the Geth, making them an enemy to all the races and not just the Quarians. An enemy who had already curb-stomped the race best in a position to put them down, and had halted their military advance, staying behind the Veil, away from known space and away from the Council.
It was not worth the Council's time to make war on the Geth. Even presuming the Quarians would pledge all their tech and loyalty forevermore (and that they would then uphold that, politics being politics), they weren't a power anymore, and any Quarian attention would be on reclaiming and rebuilding their home rather than helping the Council back. The Geth could be safely discounted as a far-off threat and/or the Quarians problem, and they could get more value by demonizing the dangers of AI (and those meddling Quarians inventing things that shouldn't have existed) from afar with the Geth as an example.
That they later shot up the Citadel at the behest of one of their own special agents leading a kind of machine-religious cult, well, that's hindsight for you.