The facts, few as they are
Joss, in an online interview which is no longer on-line, stated that the Firefly verse started as a campaign of a major Sci-Fi RPG. He's also stated that he quit playing RPG's after college. He refused to identify the system.
Since he graduated in 1986... we're looking at 1982-1986.
On to the logical speculation
The criteria "major" restricts us to games that were in commercial circulation; since his first couple years undergrad were in the UK, that restricts it further
Major Sci-Fi games in print included: GDW's Traveller & 2300, Palladium's Mechanoids trillogy & Robotech, FASA's Star Trek, FGU's Starships and Spacemen and Space Opera, TSR's Star Frontiers, and ICE's SpaceMaster.
If we eliminate the games without slugthrowers, we're left with Traveller, 2300, Mechanoids, and Robotech.
2300 was later in the era - summer 1986 - he wouldn't have been playing it during his college career.
Mechanoids lacks ship rules, and is focused on active duty games. Plus, no big bad government.
Robotech is very light on the slugthrowers, and lacks a merchantile focus.
2300, Space Opera, and Traveller have world generation rules, but only Traveller and 2300 have system generation rules.
Space Opera has everything but the right weapons.
Also, Traveller and Star Frontiers were available worldwide by 1983... and Traveller was even produced in local editions for the UK.
Which leaves, pretty much...
- Traveller or...
- something that isn't major, or...
- serious houserules from hell.
Notes in support of Traveller
I'll note as well: The Firefly class is very much, feature for feature, comparable to the specifications for the Type R Subsidized Merchant in Traveller, but with a second launch added in place of 20 tons of cargo. Deckplans differ, but still, functionally, very close.
Further, the Verse uses a high speed deep space drive, and an atmosphere capable maneuvering drive. The rest of the tech lines up really well, tho'. You need only redefine jump drive to a sub-C secondary n-space constant speed drive, and you have the ability to play the verse with stock traveller rules. Including the small vs big ship dichotomies.
Further, I've accidentally generated 5 star systems before using Classic Traveller's Book 6: Scouts; it's the only one of the lot that I recall being able to generate stellar clusters of that size; I can't get to Space Master to check.
Traveller is often derided for the "Sixguns and Starships" approach... an approach which Firefly shares.
Classic Traveller adventures often have PC's skirting local laws to do what basically amount to good deeds in the end, and/or to make enough bankroll to keep flying. Plus, the central government is often treated as the enemy in those adventures.
We can't be certain, but the most likely candidate is Traveller.