From the information they currently have, their assessment is correct.
1. It's unlikely 838-Wanda has shown her full powers
We don't know much about 838-Wanda, but from what we see, she certainly has powers - she starts to use them against 616-Wanda[*1] in the finale[*2], and she uses them to escape the Waypoint. But, she also has two children (which are real), and she didn't appear to feature in the battle against Thanos. So they probably don't think base-level Wanda is that big a threat.
Even outside 838, we've only seen hints that Wanda is very strong. In Civil War, the two sides seemed fairly balanced, and in Endgame Captain Marvel seemed to cope with Thanos about as well as Wanda did. In What If? we see Hulk fight Zombie Wanda fairly successfully. There have been hints that she is stronger than that, but nothing confirmed. I suspect 616-Captain-Marvel would feel she could take Wanda too. Certainly as a viewer, based on the Inhumans show and the X-men movies, I thought Black Bolt and Charles Xavier were going to win against her.
So, from their perspective, there's no reason to think Wanda herself is a huge threat[*3]
2. They haven't experienced the power of the Darkhold
838-Strange used the Darkhold, but it seems all he really did with it was dreamwalk to other universes - which he used to observe, rather than actively cause violence. Once he triggered an incursion, he admitted his use of it, and submitted to his friends' judgement. While the Illuminati understand the Darkhold is dangerous, they don't have experience fighting against it.
3. The Scarlet Witch appears to be more powerful than other Darkhold users
We've seen several people use the Darkhold - Agatha Harkness and several Stranges in the MCU, various others in non-MCU Marvel shows. It appears the Scarlet Witch is something different to them. She outmatches Agatha with ease, and has a shrine in the original temple. At the time this scene happens, 616-Strange, Wong, and America, are probably the only people in the multiverse who understand how powerful the Scarlet Witch actually is.
4. Black Bolt can defeat a Darkhold-wielder
The Illuminati agree to execute 838-Strange, and Black Bolt does so by merely whispering. So Reid is completely correct in what he says. All they need to do is hit Wanda once with Black Bolt's power, and she's dead. Of course, 838-Strange wasn't fighting back, but they do know defeating Wanda is possible.
5. 616-Strange is their main concern at that moment
838-Strange proved that, simply by just observing another universe, he could trigger an incursion. From what we're told (and see later), that kills (almost?) everybody in both universes - trillions upon trillions of people, nearly four times the threat Thanos posed. By merely remaining stood in front of them, 616-Strange is increasing the chances of that happening with every second. 616-Strange is trying to convince them to keep increasing the risk of an incursion, and hand over one of their most powerful artifacts to him, because of a slightly-vague threat from something 616-Strange thinks is even worse. The Illuminati's dismissal of his words seems more like an attempt to keep the focus on him, rather than get distracted by another threat.
Conclusion: The Illuminati weren't overconfident, but made a sensible assessment of the information they had
616-Strange is presenting two options: a fight against a Darhold-weilding magic user, or an incursion. The Illuminati, who have experience with both, decide the risk of the incursion is more serious than the risk from a single witch, and dismiss Strange's words. It turns out the Scarlet Witch is far more powerful than they realised,
[*1] I've seen arguments about which number the main MCU is, but 838 called it 616, so that's what I'm using here
[*2] I assumed when watching it that the climax battle happened in 838, but I've asked another question about it
[*3] I don't immediately remember if 616-Strange says its Wanda, rather than just "a witch". The prospect of Agatha Harkness attacking does not seem at all scary.