The instance in Philosopher's Stone you're referring to involve the Hogwarts school brooms. It is possible that these brooms (and perhaps all brooms) are able to respond to simple commands at a short distance.
This quote, following your own, enlightens us (emphasis mine):
"Stick out your right hand over your broom," called Madam Hooch at the
front, "and say 'Up!"'
"UP!" everyone shouted.
Harry's broom jumped into his hand at once, but it was one of the few
that did. Hermione Granger's had simply rolled over on the ground, and
Neville's hadn't moved at all. Perhaps brooms, like horses, could tell
when you were afraid, thought Harry; there was a quaver in Neville's
voice that said only too clearly that he wanted to keep his feet on the
Clearly a word-of-command has to be spoken, in this case "Up", for the broom to respond. It does not necessarily have to be wandless magic, and it is highly unlikely either way for the reasons you have given: 1) The students aren't feeling any strong emotions. 2) They're inexperienced.
Other instances of summoning brooms, all with the Accio spell:
Harry summoning his Firebolt with the Accio spell in Goblet of Fire (from dorm to Quidditch pitch)
Fred and George using the Accio spell to summon their Cleansweeps in Order of the Phoenix (from Umbridge's office to Entrance Hall)