As shown in this answer, summoned objects do not actually travel at or near the speed of light. So in terms of that aspect of safety you don't have to worry.
However, summoning objects is still not entirely safe. Consider the following case from Chapter Twenty-Nine of Order of the Phoenix:
And before Umbridge could say a word, they raised
their wands and said together, “Accio Brooms!”
Harry heard a loud crash somewhere in the distance.
Looking to his left he ducked just in time — Fred and
George’s broomsticks, one still trailing the heavy
chain and iron peg with which Umbridge had fastened
them to the wall, were hurtling along the corridor
toward their owners.
Apparently, the summoned objects don't care about things in their way. If Harry hadn't ducked, he might have been The Boy Who Was Killed By His Friends' Summoning Charms. The brooms did in fact smash at least one thing in their way:
Then there were the two large broom-shaped holes in
Umbridge’s office door, through which Fred and
George’s Cleansweeps had smashed to rejoin their
Similarly, when Sirius summoned Butterbeer in Chapter Twenty-Two the bottles plowed right through his meal that was on the table:
“That’s right,” said Sirius encouragingly, “come on,
let’s all... let’s all have a drink while we’re waiting.
He raised his wand as he spoke and half a dozen
bottles came flying toward them out of the pantry,
skidded along the table, scattering the debris of Sirius’s meal, and stopped neatly in front of the six of
In Chapter Seven of Deathly Hallows Harry was poked by a summoned object:
Harry seized the wand lying beside his bed, pointed it at the cluttered desk where he had left his glasses, and said, "Accio Glasses!" Although they were only around a foot away, there was something immensely satisfying about seeing them zoom toward him, at least until they poked him in the eye.
In addition to pulverizing innocent bystanders, summoned objects can also do damage to their intended targets. Flitwick was knocked over by parchment he summoned in Chapter Twenty-Eight of Order of the Phoenix:
“Quills down, please!” squeaked Professor Flitwick.
“That means you too, Stebbins! Please remain seated
while I collect your parchment! Accio!”
More than a hundred rolls of parchment zoomed into
the air and into Professor Flitwick’s outstretched
arms, knocking him backward off his feet.
So there definitely are occasions where summoning an object can be dangerous, either to bystanders or the caster.
However, it is not entirely clear what causes these dangerous behaviors. There are other examples where summoned objects appear to have traveled significant distances without smashing through things in their way. Harry summoned his broom from Gryffindor Tower to the dragon enclosure and there is no mention of any doors or walls being broken through as there were with Fred and George. Moreover, the broom came to a stop in front of him. In Chapter Twenty-Seven of Half-Blood Prince when Harry summoned Rosmerta's brooms, they also came to a stop, and not only that, we are told that the doors opened to let the brooms through:
"Accio Rosmerta's Brooms!"
A second later they heard a loud bang as the front door of the pub burst open; two brooms had shot out into the street and were racing each other to Harry's side, where they stopped dead, quivering slightly at waist height.