Quite simply, the tests were abandoned without everyone being tested, and there was never any indication that it could detect one of the Final Five.
The first test was a lie - Baltar wanted someone to take focus for blame away from himself, and he also wanted Aaron Doral out of the picture, and so pointed the finger at him when the Cylon device was found in ops. Later on, he "confirmed" that Doral was a Cylon, so he couldn't expose Baltar and also as further misdirection of blame - this was a guess, a lucky one as it turned out.
Baltars second detector was based around the acquisition of a nuclear warhead, which would have repercussions later on. As it turned out, this test was accurate, but slow - it would have taken Baltar years to test everyone in the fleet.
Baltars detector was declared worthless when Sharon Valerii, who had been tested and passed as human (after Baltar purposefully passed her despite a positive result), shot and nearly killed Adama, and was later proven to be a Cylon.
So, basically, he didn't know because they weren't all tested - and there was never any indication that the Final Five would have failed the test anyway, as they were not necessarily the same as the rest of the skin jobs (they created the skin jobs for the original Cylons from the 13 Colonies).