There's at least two big ones, both of which aren't apparent until later in the series, and one smaller one, which you've posited as a possibility.
The smaller one is that, yes, they were still collecting their forces. There were several episodes where that was a plot point; for example, 2x05, The Long Dark, and 3x08, Messages from Earth.
Spoiler Warning, the entire rest of this answer builds off of the Season 3 finale.
The Shadows and Vorlons stayed behind to shepherd the younger civilizations, but have drastically different ideals on how to go about it. So the younger races get used in a galaxy-wide chess match, both sides trying to use them to prove that their ideal is the correct one.
For the Shadows, conflict is what drives civilization to greater heights. They sow discord, the younger races fight each other, the strong survive and the weak rebuild stronger than before. If the Shadows openly attacked Babylon 5, they would position themselves as a strong, common enemy, that the younger races would have unite against lest they be destroyed as well - and with the Vorlons already participating in the peace talks on the station, it would likely not go well overall.
The millennia-long conflict between the Shadows and Vorlons also had its own rules, which Kosh violated when Sheridan convinced him to act directly against the Shadows, and led to Kosh's death in retaliation. If the Shadows acted directly against the station, either Kosh or Ulkesh would have been caught in the line of fire, and similar retaliation would likely have been necessary.
Lastly, it just wasn't necessary to achieve the Shadows' goals. The peace on Babylon 5 was tenuous enough as it was, all they had to do was act as mercenaries with Mr. Morden as their front man, and the Narn/Centauri war erupted anew. They only bothered to do so by the end of Season 3 because several things happened during that year that showed just how much of a threat Babylon 5 was:
- The White Star (and its superior technology) was spotted helping Earth
- The Rangers start acting openly
- Delenn convinces the Grey Council to stand with Babylon 5 against Earth (not revealed until much later to be manipulated by or working with the Shadows)
- A transport for Shadow ship seeds is hijacked, and stored at Babylon 5, which leads directly to figuring out the primary weakness of Shadow ships
The very next episode after the hijacking mentions that the Shadow ships are attacking openly, but also mentions that their targets seem random - even then, Babylon 5 isn't a target. It's not until later that they realize the whole point is to drive the survivors together and create more chaos.
Then comes War Without End, where the distress call from a possible future arrives. See my answer here for an explanation about that little bit of weirdness. But that future was averted, so an attack on the station is further delayed until the finale. (Note that this ties directly to the "gathering forces" possibility mentioned in the question)
But the events of the finale cause such upheaval that the attack gets called off - as G'Kar says, the Shadows believe that Babylon 5 is no longer a threat.