"Student Body" by F. L. Wallace. You can read the Project Gutenberg etext or listen to the X Minus one radio dramatization.
I read it in an American, English-language paperback anthology with an extraterrestrial theme, in the very late 80's or early-mid 90's,
That could have been the Asimov-Greenberg-Waugh-edited Monsters, published in 1988.
but the story itself may be older. I don't think it was too much older, though, because some of the science in it seems modern.
It first appeared it the Galaxy Science Fiction, March 1953, which is available at the Internet Archive.
The story begins with a small group of human colonists who have settled a new planet.
The first morning that they were fully committed to the planet, the executive officer stepped out of the ship. It was not quite dawn. Executive Hafner squinted in the early light; his eyes opened wider, and he promptly went back inside.
"Last night you said there was nothing dangerous," said the executive. "Do you still think it's so?"
Previous surveys of the planet show there are very few large mammals, only rat-like creatures locally and deer-like creatures that live further away,
Sort of:
The executive looked blank. Dano Marin added the explanation: "You know how Biological Control works. As soon as a planet has been discovered that looks suitable, they send out a survey ship loaded with equipment. The ship flies low over a good part of the planet and the instruments in the ship record the neural currents of the animals below. The instruments can distinguish the characteristic neural patterns of anything that has a brain, including insects.
"Anyway, they have a pretty good idea of the kinds of animals on the planet and their relative distribution. Naturally, the survey party takes a few specimens. They have to in order to correlate the pattern with the actual animal, otherwise the neural pattern would be merely a meaningless squiggle on a microfilm.
"The survey shows that this animal is one of only four species of mammal on the planet. It is also the most numerous."
and the colony's biologist suspects there was a recent environmental catastrophe that caused mass extinctions.
"Between a hundred million years and twenty thousand years ago, something happened to Glade," Marin went on. "I don't know the cause; it belongs to cosmic history and we may never find out. Anyway, whatever the cause—fluctuations in the sun, unstable equilibrium of forces within the planet, or perhaps an encounter with an interstellar dust cloud of variable density—the climate on Glade changed.
"It changed with inconceivable violence and it kept on changing. A hundred million years ago, plus or minus, there was carboniferous forest on Glade. Giant reptiles resembling dinosaurs and tiny mammals roamed through it. The first great change wiped out the dinosaurs, as it did on Earth. It didn't wipe out the still more primitive ancestor of the omnivore, because it could adapt to changing conditions.
The colonists plant their crops, but the fields are ravaged by a digging rabbit-like animal that showed up nowhere on the surveys. The biologist doesn't have an explanation, but agrees to decant the dog embryos and force-grow them to adulthood.
Close enough. Actually, the dogs are used against big rats. (It started with squirrels, then mice, then rats, then tigers . . .)
The dogs the colonists had brought had been terriers. They were still as fast, and still with the same anti-rodent disposition, but they were no longer small. It had been a difficult job, but Marin had done it well, for the dogs had lost none of their skill and speed in growing to the size of a great dane.
[. . .]
The rat tide grew in the fast crops, and the dogs were loosed on the rats. They ranged through the fields, hunting. A rush, a snap of their jaws, the shake of a head, and the rat was tossed aside, its back broken. The dogs went on to the next.
This seems to work for a while, the dogs guard the fields effectively and they see fewer and fewer rabbits, but then something starts killing the dogs. The killer turns out to be a tiger-like animal which has no precedent in the survey or fossil record. The colony's leader isn't as concerned as you might think, because it's easier to track and kill something the size of a tiger,
They stood over the animal Hafner had killed. Except for the lack of markings, it was a good imitation of a tiger. The exec prodded it with his toe.
"We chase the rats out of the warehouse and they go to the fields," he muttered. "We hunt them down in the fields with dogs and they breed tigers."
"Easier than rats," said Marin. "We can shoot tigers." He bent down over the slain dog near which they had surprised the big cat.
but the biologist has a suspicion what's going on, and goes off on an investigation to confirm it. It turns out the animal life on this planet undergoes a sort of pre-emptive Lamarckian evolution. Based on what challenges the parent animal experiences, the next generation will have characteristics to deal with it. So the rats (which, as it turns out, are the same species as the deer) gave birth to the rabbits to exploit the human crops, and when the dogs killed them, the rabbits produced the tigers to combat the dogs.
"Your facts are not straight," observed Hafner. "There are four species, ranging in size from a squirrel to a water buffalo."
"One species," Marin repeated doggedly. "They're the same. If the food supply for the largest animal increases, some of the smaller so-called species grow up. Conversely, if food becomes scarce in any category, the next generation, which apparently can be produced almost instantly, switches to a form which does have an adequate food supply."
The biologist rushes back to the colony, where he finds out that no tigers have been seen for a long time, but something new has started getting into the fields that the dogs completely ignore.
Three months before the next colonists were due, a new animal was detected. Food was missing from the fields. It was not another tiger; they were carnivorous. Nor rats, for vines were stripped in a manner that no rodent could manage.
[. . .]
Dogs were useless. The animal roamed the field they were loose in, and they did not attack nor even seem to know it was there.
He goes out and confronts this new animal - which looks very much like a human being - and thinks to himself they had better learn to co-exist peacefully with this version of the creature, because he doesn't know if they could combat what it might produce next.
"Don't you still see? There is a progression. After the tiger, it bred this. If this evolution fails, if we shoot it down, what will it create next? This creature I think we can compete with. It's the one after this that I do not want to face."
It heard them. It raised its head and looked around. Slowly it edged away and backed toward a nearby grove.
The biologist stood up and called softly. The creature scurried in the trees and stopped just inside the shadows among them.
The two men laid down their rifles. Together they approached the grove, hands spread open to show they carried no weapons.
It came out to meet them. Naked, it had had no time to learn about clothing. Neither did it have weapons. It plucked a large white flower from a tree and extended this mutely in a sign of peace.
"I wonder what it's like," said Marin. "It seems adult, but can it be, all the way through? What's inside that body?"
"I wonder what's in his head," Hafner said worriedly.
It looked very much like a man.
