Shotguns are generally slow firing firearms that fire shot, lots of small projectiles per round that spread out and cover an area. Therefore, I think shotgun protocol refers to two things.
Cover the whole area
In a normal firing team situation you would suppress fire over the enemies position and likely concentrated into specific areas. However, here they are covering the whole, albeit narrow, area to make sure the angels can't progress further than they want them too. This is also covered when Father Octavian says:
Full spread over the hostiles.
Don't do rapid fire
When fire teams are given the command to shoot at a specific target they are usually given a fire rate command with it. I can't remember specifics but I think rapid fire is pretty much as quick as you can whereas you have other rates of 1 every 3 seconds, 1 every 10 seconds and similar. Shotguns shoot slowly so it is just a "common" way of telling the viewer he wants them to shoot slower. This is again reinforced when he tells them:
We don't have bullets to waste.
He's reinforcing the fact he doesn't want them to go crazy firing consistently but to keep up a constant rate of fire. Which again is backed up by saying:
continuous fire [...] Do not stop firing while the lights are out.
If they fire too quickly they will run out of ammo and have to reload and that would be the last thing they want.
On a side note, they also have another reason to fire slowly. The muzzle flash from their rifles will, and does appear to, light up the tunnel so they can watch the Angels and so slow down their approach.
The fire team mechanics come from a British Army perspective here and things could have changed since I was there so things may be different elsewhere and nowadays.