In the novelisation for the film "Rogue One", it appears that Imperial defector Bodhi Rook is already thinking about the consequences of being a rogue (in the sense of disobeying the orders of the Rebel Council and going after Jyn Erso's father).
He thought about his passengers. Like him, they were going rogue,
courting treason by defying the Alliance council. They’d already
pilfered enough Alliance weapons and gear to supply an army; and Bodhi
had seen enough of the operation on Yavin to know equipment was in
When pushed by Rebel control, he comes up with a call sign that he thinks will pass muster.
“What’s your call sign?” the voice on the comm asked.
“Yes, we have it…” Just take off! “It’s, ah—”
Think, Bodhi. Give them something. Give them anything.
If you give them something, they might not shoot.
“—call sign Rogue. Rogue One.”
He transferred power to the thrusters, felt the familiar wobble of a cargo shuttle taking off under his control. The officer on the other end of the comm was squawking at him. Bodhi ignored it.
“Rogue One,” he declared, “pulling away!”
It would appear that the Rebels, in recognition of the great sacrifice made by Jyn and her troops started using that callsign as part of their standard repertoire afterwards, typically to designate a detachable squadron capable of carrying out specialised tasks.