People panicked because of things they saw.
Children were spinning like tops sixty feet above ground. Muggles were being insulted. Tents were being set on fire.
Two of the figures were very small. More wizards were joining the marching group, laughing and pointing up at the floating bodies. Tents crumpled and fell as the marching crowd swelled. Once or twice Harry saw one of the marchers blast a tent out of his way with his wand. Several caught fire. The screaming grew louder.
The floating people were suddenly illuminated as they passed over a burning tent and Harry recognized one of them: Mr. Roberts, the campsite manager. The other three looked as though they might be his wife and children. One of the marchers below flipped Mrs. Roberts upside down with his wand; her nightdress fell down to reveal voluminous drawers and she struggled to cover herself up as the crowd below her screeched and hooted with glee.
“That’s sick,” Ron muttered, watching the smallest Muggle child, who had begun to spin like a top, sixty feet above the ground, his head flopping limply from side to side. “That is really sick...”
Harry Potter and the Goblet of Fire, Chapter 9, The Dark Mark
And finally the dark mark appeared, the symbol of Lord Voldemort.
For a split second, Harry thought it was another leprechaun formation. Then he realized that it was a colossal skull, comprised of what looked like emerald stars, with a serpent protruding from its mouth like a tongue. As they watched, it rose higher and higher, blazing in a haze of greenish smoke, etched against the black sky like a new constellation.
Suddenly, the wood all around them erupted with screams. Harry didn’t understand why, but the only possible cause was the sudden appearance of the skull, which had now risen high enough to illuminate the entire wood like some grisly neon sign. He scanned the darkness for the person who had conjured the skull, but he couldn’t see anyone.
“Who’s there?” he called again.
“Harry, come on, move!” Hermione had seized the collar of his jacket and was tugging him backward.
“What’s the matter?” Harry said, startled to see her face so white and terrified.
“It’s the Dark Mark, Harry!” Hermione moaned, pulling him as hard as she could. “You-Know- Who’s sign!”
Harry Potter and the Goblet of Fire, Chapter 9, The Dark Mark
The Dark Mark generally meant somebody was murdered!
“And it hasn’t been seen for thirteen years,” said Mr. Weasley quietly. “Of course people panicked... it was almost like seeing You-Know-Who back again.”
“I don’t get it,” said Ron, frowning. “I mean... it’s still only a shape in the sky...”
“Ron, You-Know-Who and his followers sent the Dark Mark into the air whenever they killed,” said Mr. Weasley. “The terror it inspired... you have no idea, you’re too young. Just picture coming home and finding the Dark Mark hovering over your house, and knowing what you’re about to find inside...” Mr. Weasley winced. “Everyone’s worst fear... the very worst.”
Harry Potter and the Goblet of Fire, Chapter 9, The Dark Mark
When Voldemort actually returned, Harry saw it. Nobody else saw and the ministry went into denial.
“Voldemort doesn’t march up to people’s houses and bang on their front doors, Harry,” said Sirius. “He tricks, jinxes and blackmails them. He’s well-practiced at operating in secret. In any case, gathering followers is only one thing he’s interested in. He’s got other plans too, plans he can put into operation very quietly indeed, and he’s concentrating on those for the moment.’
Harry Potter and the Order of the Phoenix, Chapter 5, The Order of the Phoenix
People thought Voldemort was back when they saw his symbol, the Death Mark. But when he actually returned most folks did not see it and did not believe Dumbledore's words because it was easier to be on denial.