Species 8472 posed a much greater threat than the Borg.
While the Borg had a clear cut tactical advantage over the Federation (and all other Alpha/Beta powers), it was still possible for Federation starships to withstand Borg weapons for a limited period of time and potentially retaliate. The battle of Sector 001 is a great example of this: yes, Federation ships were getting destroyed left and right, but we see ships staying alive long enough to take shots at the cube and inflict damage. Species 8472 Bioships can destroy Borg cubes in a single shot, and with a few more ships they can do the same to whole planets--destruction on a scale never seen from the Borg. While I'm sure the Borg could destroy planets if they put their collective mind to it, Species 8472's firepower superiority was painfully obvious from the first ten seconds of Scorpion Part 1.
Regardless of either race's destructive capabilities, Species 8472 and the Borg had fundamentally different approaches to galactic conquest. Aside from the occasional 1-cube incursion to the Alpha Quadrant, the Borg mostly stayed in the Delta Quadrant and posed very little immediate threat to the Federation. Also, since they desired to assimilate all beings they encountered, it was in their best interests to not undertake actions that would destroy the Federation's inhabitants outright (such as blowing up the earth). Species 8472 on the other hand had no desire or need to keep any of the galaxy's species alive. Indeed, since they were the only intelligent life from Fluidic Space, it's presumable that they already finished off everyone else in their home dimension, assuming other species ever evolved in the first place.
Why Species 8472 were so aggressive is never established on screen, but I believe we can infer some hints as to their motivation from what little communication with them took place. Their goal of wiping out the Borg is perfectly understandable; the Borg attacked them and Species 8472 retaliated. However, this does not explain their aggression towards the rest of the galaxy.
While Janeway never communicated with Species 8472 directly, Kes had psychic contact with them that firmly established their murderous intentions. I believe their exact word were "The weak will perish" (full quote in Izkata's answer) and "Your galaxy will be purged." This implies a "survival of the fittest" doctrine embedded deep within their culture; they probably saw our galaxy's life as inferior, and judged that it deserved to be extinguished.
So you have the choice of allowing the Borg to rest in the Delta Quadrant as they have for centuries or allowing a new, highly malicious race to crusade through the galaxy destroying all life that it encounters. In my opinion, Janeway's choice was clearly correct.