This is addressed in the novelisation which contains a few extra lines of dialogue. In short, the Guardians are zealots. Although their cause is roughly aligned with the Rebellion, they're unreliable and likely to bring down Imperial attention.
She frowned. “What kind of trouble?”
Cassian turned his head in a slow arc as if checking for pursuit. “For
the Guardians, anyone who’s not a pilgrim is intruding on holy ground.
The Empire calls them strays…used to be domesticated, still beg for
scraps, but they’ve really gone feral. Look at them wrong and they’ll
bite your hand in a second.” “You’ll make me like them,” Jyn said.
She tried to push their faces, Chirrut’s voice, out of her brain. They
probably were con men, even if they’d been zealots once. Beyond that,
she didn’t know enough about the local religions to speculate;
pilgrims from a hundred faiths came to the moon from across the
galaxy, and all of them blurred together into the same pathetic cult,
chanting and moaning and squirming under the Empire’s boot.
Rogue One: A Star Wars Story