The term Rogue Squadron is a relatively widely known term in the Star Wars universe today, especially with the movie Rogue One coming out soon. It makes me wonder though, if there are criteria to meet in order for the group of X-wings to be dubbed Rogue Squadron. Is this only the term given to those X-wing pilots in this certain squad, or is it the rebel term for any X-wing unit currently dispatched in combat? I know we see reference to Red Squadron too, so it may be connected to wherever the units are stationed. Does it require a specific number of X-wings? I'm not certain where to go for the answer to this, can anyone help to answer these questions?
To my knowledge there isn't currently an accepted Canon answer about the origin or history of Rogue Squadron. However, a squadron and it callsigns do not appear to be ad hoc things assigned to whichever unit is out fighting. Squadrons were permanent units that were stationed, trained, and operated together. Known squadrons include:
In the Rebels TV series, we meet Phoenix Squadron, a squadron of A-Wing fighters that operates as part of the Phoenix cell.
Gold Squadron operated BTL Y-Wing bombers. They are known to have fought at the Battle of Scarif and the Battle of Yavin, and to have some continuity of personnel between those battles.
Operated X-Wing fighters in the Battles of Scarif, Yavin, and Endor. The same person commanded as Red Leader at both Scarif and Yavin, and Wedge Antilles, one of only two members to survive the Battle of Yavin, identified as Red Leader in the Battle of Endor.
For such an important group in Legends and the fandom, Rogue Squadron's appearances in canon are extremely limited. They only show up in one film - Empire Strikes Back - piloting not X-Wings, but snowspeeders. They're the pilots who find Luke and Han when they're stuck outside overnight, and the pilots who fly air support against the AT-ATs. As far as the films go, that's their only appearance.
So, while we don't have extensive canon about Rogue Squadron as a unit, it does appear that squadrons in the Rebel Alliance were permanent, formal units, but also that there is some overlap of personnel between squadrons (Wedge Antilles and Luke Skywalker were both members of Red Squadron and Rogue Squadron).
In Legends, we have a lot more information to go on, thanks especially to the X-Wing novels by Michael Stackpole. It depicts Rogue Squadron as the New Republic's elite starfighter squadron, with pilots hand-selected and trained by Wedge Antilles. They have an extensive combat record, both in space combat and ground-based special operations; their personnel infiltrated Coruscant and were pivotal in bringing down its planetary shields to allow the Republic to capture the world, and they later went 'rogue' and fought a war all by themselves.
The Rebel Alliance named their fighter squadrons after things that symbolized their rebellion. That's why you have squadrons like
Phoenix Squadron (
canon from Star Wars Rebels) which represents the rise from the (eventual) ashes of the empire into a new republic.
Rogue Squadron was named Rogue since it was a hastely assembled rag-tag band of rebels without much of any experience (as opposed to professional pilots like
Poe Dameron later. Rogue squadron consisted of 12 pilots before the battle of Yavin (only Luke and Wedge survived).
Once the Rebellion turned into a fully fledged galactic civil war, the squads became more formalized and oranized and the squad was eventually renamed to Red Squadron (that's why you have squads like gold squadron etc later on).