The agents are portrayed as CIA/FBI agents in the Matrix-human world. In the first scene of The Matrix, they seem to have authority over the police. This wasn't accomplished by taking over their bodies. It seems they play a real bureaucratic role. What do they do on a daily basis when they don't have anomalies like Neo and Trinity to hunt down? Do they actually hold a desk job, go in to work every day and solve crime to keep up pretenses?
Agents are responsible for protecting the integrity of the Matrix from both external and internal threats. Zion operatives like Neo and Trinity are external threats because they hack into the Matrix and their actions can reveal the truth that the Matrix is not the real world. However, there are also internal threats which can reveal the truth about the Matrix: in particular, Exile programs.
The Oracle: Look, see those birds? At some point a program was written to govern them. A program was written to watch over the trees, and the wind, the sunrise, and sunset. There are programs running all over the place. The ones doing their job, doing what they were meant to do, are invisible. You’d never even know they were here. But the other ones, well, we hear about them all the time.
Neo: I’ve never heard of them.
The Oracle: Of course you have. Every time you’ve heard someone say they saw a ghost, or an angel. Every story you’ve ever heard about vampires, werewolves, or aliens is the system assimilating some program that’s doing something they’re not supposed to be doing.
Neo: Programs hacking programs. Why?
The Oracle: They have their reasons, but usually a program chooses exile when it faces deletion.
Neo: And why would a program be deleted?
The Oracle: Maybe it breaks down. Maybe a better program is created to replace it – happens all the time, and when it does, a program can either choose to hide here, or return to The Source.
The Matrix Reloaded (transcript)
Programs which have broken down and choose exile instead of voluntary deletion constitute a threat since they are not under the control of the system and do not obey the rules of physics that blue pills in the Matrix are familiar with. Agents are tasked with neutralizing the threat posed by these exile programs. That's why the Agents considered the exile program the Keymaker a higher priority than even Trinity and Morpheus during the freeway chase:
Agent Johnson: The exile is the primary target.
Keymaker: We do only what we’re meant to do.
Agent Johnson: Then you are meant for one more thing. Deletion.
The Matrix Reloaded
It is also noted by Smith that he and other Agents had difficulty attempting to capture Seraph back when Smith was an Agent and Seraph was an exile:
I remember chasing you was like chasing a ghost.
The Matrix Revolutions (transcript)
Exile programs need not have a human appearance. In The Animatrix episode Beyond we see a haunted house in which defective programs cause the house to exhibit anomalies which break the physics of the Matrix. At the end of the episode Agents are seen supervising the destruction of the house so blue pills cannot stumble across it and begin to question the Matrix:
Source: http://www.intothematrix.com/rl_img/anime_gallery_beyond_5L.jpg. The video of the destruction of the house can be seen in this clip on Youtube.
Between external threats (Zion operatives) and internal threats (exile programs), Agents had plenty to do to keep busy.