There may be a number of stories fitting this description, it doesn't sound like an uncommon theme to me. Here is one possibility.
The Rodent Laboratory by Charles Platt.
It concerns a large experiment in an isolated research centre:
Harris stood on the catwalk in the darkened chamber, leaning against
the railing, staring down into the brightly lit test area below him.
The rats were in one of their more active states: brown shapes
wriggled and scuttled over the thinly-sanded flooring. A group of
them huddled round the feeding troughs, jostling for the best
position. A male chased a female into one of the breeding hutches. A
mother crouched in one corner of the enclosure, suckling young rats
only two or three days old, baring her fangs at any intruder. The
population had increased now to the point where there was no longer
any room in the hutches for females to rear their young.
The test area is viewed from behind one-way glass:
On duty in the observation chamber that
evening, Harris sat in the darkened room with Carter, watching and
noting developments as they occurred. The silence in the place was
overpowering. Under the one-way glass, the rats scuttled about,
oblivious of the men watching in the darkened area above.
The rats indulge in cannabalism depite there being ample food.
"Some of the mothers have eaten their young straight after the
"Look," he said, suddenly animated, "the large
one, there, by the feeding trough." As they watched, the large rat
threw itself at a smaller one, dragging it by the neck, kicking up the
sanded flooring. It bit viciously, and the smaller one twitched and
lay still. The large rat eagerly seized its place at the trough.
"Interesting," he said. "That's been happening
more and more often. Wait, now. Here come the scavengers." Thin,
nervous-looking rats sidled up to the corpse of the victim and began
dragging it away, chewing at it.
The experiment has parallels in the human world of the research establishment.
"You can tell me how I fit nine people into eight rooms. There are two
new shorthand typists here, for report work, I suppose. There are
several journalists from the science magazines, who'll have to stay
overnight. More additions to the lab staff ... Philip, who's been
bringing all these people here? We're overcrowded enough as it is."
The rats manage to establish some sort of hive mind and engineer their