"2430 A.D." by Isaac Asimov. I first read it in Buy Jupiter and Other Stories, a collection of several of his shorter pieces of SF. I've just looked it over again now, and the plot is a close match for what you summarized from memory. Two men representing the government prevail upon the last holdout to "voluntarily" consent to the destruction of the last collection of small animals (including, as you said, a few mice) which his family has long had permission to keep as pets.
One other thing, though -- the way you phrased it, you made it sound as if, by the end of the story, there were literally no living things on Earth except humans. Aside from the minor point of microorganisms (such as the bacteria that live in the human body and aid in our digestion, which I admit aren't mentioned in Asimov's story), you left out the subject of providing food for all those humans! In the text, it is stated that the oceans have been turned into "plankton soup" in order to feed everybody who lives on dry land.
(I'd think that would get to be an awfully monotonous diet if you were never served anything else, but perhaps they had a wide range of artificial flavorings to put variety in their daily meals? The story doesn't go into detail about this point; it just briefly mentions the plankton as the only food crop left in the world.)
Here's an excerpt from the conversation that takes up much of the story. It describes how the decision had been made to end up with a balanced population of fifteen trillion human beings (or two trillion tons of human biomass, which amounted to the same thing). The plan was that from now on the birth rate and the death rate were expected to stay in perfect equilibrium, since they knew they had reached a practical limit.
“It was known in advance exactly how many men the Earth could support.
So many calories of sunlight reached the Earth, and, using that, only
so many tons of carbon dioxide could be fixed by green plants each
year, and only so many tons of animal life could be supported by those
plants. The Earth could support two trillion tons of animal life--”
Cranwitz finally broke in, “And why shouldn't all two trillion tons be
“Even if it meant killing off all other animal life?”
“That's the way of evolution,” said Bunting angrily. “The fit