One assumes that Hogwarts has a muggle-repelling charm cast upon it.
‘But Hogwarts is hidden,’ said Hermione, in surprise, ‘everyone knows that … well, everyone who’s read Hogwarts: A History, anyway.’
‘Just you, then,’ said Ron. ‘So go on – how d’you hide a place like Hogwarts?’
‘It’s bewitched,’ said Hermione. ‘If a Muggle looks at it, all they see is a mouldering old ruin with a sign over the entrance saying DANGER, DO NOT ENTER, UNSAFE.’
‘So Durmstrang’ll just look like a ruin to an outsider, too?’
‘Maybe,’ said Hermione, shrugging, ‘or it might have Muggle-Repelling Charms on it, like the World Cup Stadium. And to keep foreign wizards from finding it, they’ll have made it Unplottable –’
Harry Potter and the Goblet of Fire: — CHAPTER ELEVEN —
Aboard the Hogwarts Express
We see how these work earlier in the book.
‘Seats a hundred thousand,’ said Mr Weasley, spotting the awestruck
look on Harry’s face. ‘Ministry task force of five hundred have been
working on it all year. Muggle-Repelling Charms on every inch of it.
Every time Muggles have got anywhere near here all year, they’ve
suddenly remembered urgent appointments and had to dash away again …
Bless them,’ he added fondly, leading the way towards the nearest
entrance, which was already surrounded by a swarm of shouting witches
and wizards.
Harry Potter and the Goblet of Fire: — CHAPTER EIGHT — The Quidditch
World Cup