Arguably it's a movie maker's point rather than a plot point. They need to distract from the fact that the trace has just found the location and that if they were paying attention rather than bickering, they could already be making strides towards resolving it but as they are more concerned about Banner with the scepter it's missed. This sets up realization this was Loki's plan and a moment of calm before the bomb goes off. It's a false dawn because of course Hulk then manifests himself moments later as intended, but the job was done, their eyes were off the ball at the crucial moments and on Banner.
His reaction by putting it down shows that he knew immediately he'd been played and he was back "in the room" and in control. The others though had seen the glimmer, so even had the attack not happened the damage was arguably done. To me it was that he reacted again to being "asked nicely" rather than a conscious decision. Cap asks him nicely to put the Scepter down and he does, just as Widow "asks nicely" for him to come in and even the little girl asking nicely for him to come to the sick person. It seems Banner is so used to no contact that basic courtesy shown to him makes him react favorably to them, even Stark's joking "suiting up" stuff is nice rather than goading.
One thing is sure, it's not the Scepter itself, but it perhaps amplifies Loki's powers. Banner is immediately showing signs of Loki "in his head"/migraine when he even sees Loki sans staff... the Scepter probably just allows Loki to be "in that room" even when he is decks away.