All evidence is that Saruman had been building his army for a good many years prior to Gandalf's capture.
In the Council of Elrond we see that the industrialization of Isengard was already at an advanced stage by the time Gandalf was placed on top of the tower:
They took me and they set me alone on the pinnacle of Orthanc, in the place where Saruman was accustomed to watch the stars. There is no descent save by a narrow stair of many thousand steps, and the valley below seems far away. I looked on it and saw that, whereas it had once been green and fair, it was now filled with pits and forges. Wolves and orcs were housed in Isengard, for Saruman was mustering a great force on his own account, in rivalry of Sauron and not in his service yet.
A possible date for this is given in the Tale of Years:
c. 3000. The shadow of Mordor lengthens. Saruman dares to use the palantír of Orthanc, but becomes ensnared by Sauron, who has the Ithil Stone. He becomes a traitor to the Council. His spies report that the Shire is being closely guarded by the Rangers.
Note 7 to the Palantíri essay in Unfinished Tales gives further evidence of an earlier beginning (with my emphasis):
The Council seems to have been unaware, since for many years Isengard had been closely guarded, of what went on within its Ring. The use, and possibly special breeding, of Orcs was kept secret, and cannot have begun much before 2990 at earliest. The Orc-troops seem never to have been used beyond the territory of Isengard before the attack on Rohan.
So to summarize:
Saruman had been building his army for possibly up to 30 years prior to Gandalf's capture, it wasn't the case that Gandalf was spectacularly unobservant, Saruman had actually kept this secret by breeding the Uruk-Hai in caverns underneath Isengard and not dispatching them outside of his territory. Saruman not only had the trust of everyone but the lush gardens of Isengard helped provide the perfect camouflage, concealing the army up until Gandalfs visit and imprisionment, it was only after Gandalf's capture that Saruman was open about his activities.