Unknown where exactly he got it from but the Skrulls has been on Earth for a while at this point. To the point where this one in particular had managed to slip in alongside Fury as Coulson without anyone else noticing. It’s not implausible for the Skrull to have gotten a pistol from any number of sources: a store, stolen from someone, pinched off another officer or even got it out of the glovebox in the car. Most important cars for agents would have pistols hidden away in them in case they lose their personal one.
In short it doesn’t really matter where or how he got it though. It’s not important to the story and it would have been odd for him not to have one whilst disguised as Coulson.
FWIW the Skrull impersonating Coulson appears as Fury is interrogating Vers at the phone before the other Skrull shoots from the rooftop. We know this because there’s a small time skip from Coulson talking to the security guard and then approaching Vers. In that time, Coulson would have entered the Blockbuster, and whilst in there the Skrull took his place and approaches from behind Fury during the conversation. This same Skrull as Coulson is the one that gets in the car with Fury.