"The Easy Way Out" by "Lee Correy", pseudonym of G. Harry Stine (ISFDB, Wikipedia).
I'm trying to identify a story about scouts for an alien invasion. The aliens have a weapon that will kill any creature with a nervous system advanced enough to be intelligent, and they believe that the most ferocious creatures are always the dominant intelligence.
"There is no obvious reason why we can't take over this planet for the Great Overlord."
"There may be several reasons why we can't. It all depends on the native life forms, particularly with regard to their Intelligence Index, Adaptability Index, and, most important, Ferocity Index."
The first creature that they encounter is a grizzly bear, and of course their weapon does not work and the bear kills several scouts.
Very few native life forms will bother a feeding grizzly bear. Ursus horribilis is not only strong, but easily provoked. But Harmarrght didn't know this. He found out quickly.
He fired an energy bolt at what should have been an area of vital control in the bear's midsection with the intent to paralyze the beast. The shot singed white-tipped hair and burned a hole through the skin. It hurt the bear and drew its attention to the young alien.
"Cover him!" Norvallk yelled to his party. One of the warrior techs burst forward to get between Harmarrght and the bear.
The bear stopped fishing and let out a bellowing roar. This panicked the warrior tech who fired a projectile toward the bear's head. Another of the warrior techs got into position. But the bear moved . . . fast.
The grizzly brought its huge forepaw down on the closest warrior alien. Armor and all, the warrior splattered.
The next swipe of the huge paw demolished Grahhgh, who had the misfortune to get within range. While trying to get to the second warrior, the bear stepped on Rastharrh, putting part of him out of commission. The bear rose on its hind legs to its full height of eight feet and started to swing again, aiming toward Harmarrght, but the second warrior fired an explosive bolt that caught the grizzly in the roof of the mouth and congealed its brain.
They decide that the bear must be so advanced that he can resist their weapon and they follow at a distance.
Not the same bear:
Two sunrises later, the party discovered another silvertip grizzly. The aliens didn't repeat their first mistake; they stood well back and watched this bear closely.
The bear then encounters and loses a fight to a wolverine.
The wolverine simply attacked the bear as though it did not know the meaning of fear. Its flashing teeth and slashing claws were smaller and less strong than the bear's, but sheer meanness was on its side. It ripped in to kill, giving no quarter. After the first encounter in which the bear's huge paw missed in a roundhouse swing, the battle was very short and very one-sided.
The grizzly took the easy way out. It retreated, ambling off into the pine forest as rapidly as it could move.
This convinces them that the bear wasn't the dominant species after all, and that the wolverine must be even more advanced.
The computer chewed up the available data regarding size, probably body mass, and other related factors of the two different animals, bear and wolverine; it then compared this with data from other worlds, considered the possibility of reducing the high Ferocity Index of he bear, found that it could not logically do so, discovered that it could not handle the Ferocity Index of the wolverine, and ended up with a stoppage. The wolverine's Ferocity Index was off-scale.
The wolverine turns out to be a pet though, and the two children who scold it for being away so long have the aliens really afraid, as they are obviously dominant over the wolverine. Of course, then the children's parents call them into the house for dinner.
The young bipeds talked to it, played with it, and fondled it for some time. The aliens recorded every movement and sound. Norvallk was very busy trying to make things add up in his mind; he was quite unhappy with the conclusions he was reaching. Harmarrght merely watched in great confusion; he was having great difficulty rationalizing what his own logic told him with what he had been taught.
A large biped appeared in the cabin door. "Boys! Lunch time! Come in now!"
They started to go, but the wolverine wanted more play and love. It growled and tried to nip at one boy's leg.
The human child turned around and cuffed the wolverine smartly, scolding it as he did so.
The wolverine shook its fur and followed the boys into the cabin.
The aliens decide to leave, as they realise that the ones dominating so ferocious a beast are only juveniles.
"But we could certainly overcome . . ."
"Again, my apologies. Do you think we could fight the several life forms we saw on that planet without expending millions of warriors and a great deal of equipment? Remember the universal law of living organisms: the Law of Least Effort. This is a big galaxy, and there are more comfortable and less expensive parts of it in which to operate.
"I guess you're right," Harmarrght admitted. "There are easier things to do."