You're making a few leaps in your assumptions:
I can see how during the first contact they'd slay the astronauts... "Turning off their radio" as it were. They could reasonably think they weren't hurting anybody, because drones are like fingernails. At this point they must have thought they could eventually contact the humans' queen.
It makes no sense to kill the astronauts to "turn off the radio" and then try to contact the human queen. If they wanted to contact the human queen and thought that the astronauts were psychically connected to the queen, they could've used the astronauts.
It makes much more sense that they simply saw the astronauts react differently to their presence (e.g. one panics, the other remains diplomatic, the third one becomes hostile). Their different approaches directly proves that humans are not a hive mind, thus confirming their notion that humans are not intelligent (as they already assume non-hive minds to be incapable of intelligence).
But during the Second Invasion ... the humans attack them with sophisticated tactics. They shoot them with nuclear missiles, from spaceships. This can't possibly be instinctive behavior. They had to have realized that there was an intelligence running the humans.
If you know how evolution works, then you should see that complex tactics are not always proof of intelligence.
Giraffes didn't always have long necks. No giraffe has ever "thought" to extend it's own neck. There was never an intelligent thought by a giraffe that contemplated how to achieve a longer neck.
However, giraffe's necks still grew over time. Because the giraffes who just happened to be born with longer necks, also ended up as being more succesful (because they could reach more leaves than their short-necked brothers), thus making them more likely to procreate and pass their genes on.
Over centuries, giraffe necks increased in length. This cleverly solved a problem in their environment: being able to reach as many leaves as possible. However, this is not a sign of intelligence.
The same is true of ants. Suppose that ants are not always born with an innate sense of group spirit (living for the colony instead of yourself). Let's say there is a 50/50 coin flip whether an ant is selfish or selfless (for the sake of example).
We can already conclude that the ants are not intelligently deciding to be selfless or selfish. It is simply their genetic makeup that invariably drives their behavior. The ants are not intelligent.
So let's say 100 ants exist, 50 selfish ants and 50 selfless ants.
The selfish ants live by themselves. Even if they try living together, their selfish nature will eventually break any alliance that they have.
The selfless ants, however, start working together, and their alliance does not break (because they all observe their common shared ideal, and don't resort to selfish behavior).
As it turns out, a colony is much stronger than a sole ant, and after a year many more selfish ants have died (compared to the selfless ants).
If this same pattern repeats itself time and time again, then the selfless ants are more likely to survive and procreate. Over time, this means that the ant population disproportionately favors selflessness over selfishness (genetically speaking).
Over centuries' time, the ants will be selfless. But they did not bcome selfless intelligently. It happened evolutionarily.
The humans' tactical approach may simply be a consequence of their genetic makeup. Centuries of evolution may have proven that humans who do have tactics are more successful (evolutionarily speaking) than humans who do not have tactics; thus making "tactically inclined" humans genetically predominant over a long period of time.
Your assumption that human tactics prove human intelligence are not necessarily correct. Human tactics could also have emerged as a long-term consequence of evolution.
Note that I only addressed human tactics, but the same principle applies to everything else, including the fact that humans use weapons.
- So did Neanderthals. But someone alive in 2017 doesn't consider a Neanderthal as intelligent. If the Buggers are sufficiently advanced, they will consider human weaponry as "laughably simplistic", and not as an indication of intelligence.
- The weapons may have been developed by another species, and humans have gotten their hands on them before their civilization reached a point where they're intelligent enough to build them themselves (like a child who finds a gun and plays with it).
- Many animals make their own weapons (venom), but their bodies automatically (genetically) produce this weapon. The animal does not intelligently know how to make venom.
Could the Buggers have realized that humans are intelligent?
Yes, you are right about that.
Should the Buggers have invariably realized that humans are intelligent?
Not necessarily. The indications of intelligence that they "should have seen" are not irrefutable evidence. There are explanations that can forgo the acknowledgement of human intelligence.
Furthermore, it stands to reason that the Buggers' notion that "non-hive-minds cannot possible be intelligent" inherently prejudiced the Buggers to assume that humans cannot possibly be intelligent regardless of wielding weaponry and tactics.
Consider what a (figurative) echo chamber is:
In news media, the term echo chamber is analogous with an acoustic echo chamber, where sounds reverberate in a hollow enclosure. An echo chamber is a metaphorical description of a situation in which information, ideas, or beliefs are amplified or reinforced by communication and repetition inside a defined system. Inside a figurative echo chamber, official sources often go unquestioned and different or competing views are censored, disallowed, or otherwise underrepresented.
I would argue that a hive mind is inherently an extreme echo chamber. A hive mind only has one opinion, so they are not experienced in how to handle conflicting ideas. The same wrong notion ("non-hive-minds cannot be intelligent") gets repeated over and over, to a point where it is assumed as a common truth, and no longer as an unproven assumption or a rumor.
And once you accept something that common truth, then you won't easily be swayed to believe the opposite. So it's perfectly possible that the Buggers never reevaluated their prejudices, even if they could've noticed human tactics and weaponry.