Like, if they met someone for the first time, would they be able to tell whether the person is a wizard/witch or a Muggle?
No they can't.
Tom Riddle, an exceptional wizard, did not recognize that Dumbledore was a wizard when the latter came to meet him.
“You can’t kid me! The asylum, that’s where you’re from, isn’t it?
‘Professor,’ yes, of course — well, I’m not going, see? That old cat’s the one who should be in the asylum. I never did anything to little Amy Benson or Dennis Bishop, and you can ask them, they’ll tell you!”
“I am not from the asylum,” said Dumbledore patiently. “I am a teacher and, if you will sit down calmly, I shall tell you about Hogwarts. Of course, if you would rather not come to the school, nobody will force you —”
“Are you a wizard too?”
“Yes, I am.”
“Prove it,” said Riddle at once, in the same commanding tone he had used when he had said, “Tell the truth.”
Harry Potter and the Half blood Prince
So that implies that Wizards can't know wizards just by looking at them or sensing some sort of Aura or something. They can however detect if a place has known magic or not, as Dumbledore did in that cave in the Half blood prince.
Of course Tom was not a trained wizard at that time so consider these examples. The rumors about Ariana Dumbledore, for instance. She was a witch yet rumors flew about her being a squib (And yes, a neighbor, Bathilda Bagshot I think, did see her).
Neville's relatives also thought he was a squib before he accidentally did a bit of magic.
My great-uncle Algie kept trying to catch me off my guard and force some magic out of me – he pushed me off the end of Blackpool pier once, I nearly drowned – but nothing happened until I was eight. Great-uncle Algie came round for tea and he was hanging me out of an upstairs window by the ankles when my great-auntie Enid offered him a meringue and he accidentally let go. But I bounced – all the way down the garden and into the road. They were all really pleased. Gran was crying, she was so happy. And you should have seen their faces when I got in here – they thought I might not be magic enough to come, you see. Great-uncle Algie was so pleased he bought me my toad.’
Harry Potter and the Philosopher’s Stone
These mistakes were made by fully grown, qualified wizards. If they could tell just by looking at the other person, surely these mistakes wouldn't have happened.
Some other examples:
- Harry's failure at determining that Arabella Figg was a squib.
- Harry's and apparently other students' failure at determining that Filch was a squib as well.
As mentioned by Harry Johnston and I quote:
In Deathly Hallows, Harry, Ron and Hermione didn't identify the Death Eaters in the coffee shop as wizards until they pulled their wands. (But they weren't paying attention to them, which might make a difference.)
It doesn’t seem like wizards could ‘sense’ a wizard from a Muggle.
Wizards don’t seem to have a natural ability to ‘sense’ whether someone is a wizard or a Muggle. They’ve come up with intentional signs, like wearing both purple and green, to indicate to their fellow wizards around that they too are magical. If they were able to somehow ‘sense’ the magical ability in other wizards, they’d have no need for these methods of ‘signaling’ to fellow wizards.
But by certain spells, signs or magical creatures, they could.
They can figure it out, however. While the absence or presence of magic isn’t something wizards can sense, it is indeed detectable in certain ways.
Casting a Muggle-Repelling Charm on an area they have to pass through (like a doorway for example) could work, since if they’re a wizard it won’t affect them, but if they’re a Muggle, they’ll suddenly remember a reason they need to leave and they’ll walk away.
“Muggle-Repelling Charms on every inch of it. Every time Muggles have got anywhere near here all year, they’ve suddenly remembered urgent appointments and had to dash away again …”
- Harry Potter and the Goblet of Fire, Chapter 8 (The Quidditch World Cup)
The enchantment on Hogwarts can make Hogwarts look like a ruin to Muggles, so it can ‘tell’.
“It’s bewitched,’ said Hermione. ‘If a Muggle looks at it, all they see is a mouldering old ruin with a sign over the entrance saying DANGER, DO NOT ENTER, UNSAFE.”
- Harry Potter and the Goblet of Fire, Chapter 11 (Aboard the Hogwarts Express)
This also means that it’s possible for spells to tell the difference between wizards and Muggles, so some enterprising wizards (like Fred and George perhaps) could possibly invent a Muggle detector.
From a Pottermore writing, the quill and book that choose who’ll be invited to Hogwarts can tell the difference between wizards, Muggles, and Squibs, who are somewhat between Muggle and wizard.
Those few who have observed the process (several headmasters and headmistresses have enjoyed spending quiet hours in the Book and Quill’s tower, hoping to catch them in action) agree that the Quill might be judged more lenient than the Book. A mere whiff of magic suffices for the Quill. The Book, however, will often snap shut, refusing to be written upon until it receives sufficiently dramatic evidence of magical ability.
* - The Quill of Acceptance and The Book of Admittance (Pottermore)*
It’s possible that other enchanted objects can detect the difference as well.
There are some magical creatures that can tell the difference between wizards and Muggles. In some cases, wizards can use the knowledge of these creatures to their own advantage. Crups were likely bred by wizards for this exact purpose, and can be kept as pets, so wizards could get their own Crup to use as a Muggle detector once they can control it.
“The Crup is almost certainly a wizard-created dog, as it is intensely loyal to wizards and ferocious towards Muggles. It is a great scavenger, eating anything from gnomes to old tyres. Crup licences may be obtained from the Department for the Regulation and Control of Magical Creatures on completion of a simple test to prove that the applicant wizard is capable of controlling the Crup in Muggle-inhabited areas.”
- Fantastic Beasts and Where to Find Them
Nagini was somehow able to detect that Frank Bryce was a Muggle.
“Indeed, yes,’ said the voice. ‘According to Nagini, there is an old Muggle standing right outside this room, listening to every word we say.”
- Harry Potter and the Goblet of Fire, Chapter 1 (The Riddle House)
There may be other magical creatures that can tell wizards from Muggles, but these are confirmed.
Unintentional behavioral cues:
Wizards and Muggles react differently to certain things. For example, wizards would understand terms like ‘Quidditch’, and Muggles would not. Muggles would also react surprised or in disbelief to seeing ‘strange’ things that wizards would recognize as the work of other wizards. Harry was able to tell a man was a Muggle by his behavior. In this case, it was likely his reaction to the tents, many of which were noticeably ‘enhanced’, which would be unusual to Muggles but normal to wizards.
“A man was standing in the doorway, looking out at the tents. Harry knew at a glance that this was the only real Muggle for several acres.”
- Harry Potter and the Goblet of Fire, Chapter 7 (Bagman and Crouch)
Observant wizards could use these cues to tell wizards from Muggles.
Intentional signs to other wizards:
Signs that wizards intentionally use to indicate to others of their kind that they’re a wizard are mentioned in Pottermore writings. For example, wizards often wear a combination of purple and green as a clue to other wizards.
Witches and wizards often reveal themselves to each other in public by wearing purple or green, often in combination.
- Colours (Pottermore)
In addition, discussions of the Scottish rugby team is another way for wizards to subtly identify each other, and this writing states that there are other ways that wizards have developed as well.
Over the nearly 150 years since Angus Buchanan helped win the first international rugby match, discussing Scottish rugby has become one of several covert identifiers for wizards meeting in front of Muggles and seeking to establish each other’s credentials.
- Scottish Rugby (Pottermore)
The existence of these methods in itself implies that wizards can’t instinctively sense other wizards, as if they could, covert identifiers would be unnecessary since they’d know instinctively.
No, they can't.
One way that wizards determine who another wizard is by asking another who their favorite rugby team is. Wizards always answer "Scotland", Muggles do not. I don't think they'd need this covert identification if they could tell otherwise.
Yes, they could
Even Muggles can distinguish wizards from muggles but probably will not name them as wizard, rather a weirdo.
"Ah - right - certainly -" said Mr. Weasley. He retreated a short distance from the cottage and beckoned Harry toward him. "Help me, Harry," he muttered, pulling a roll of Muggle money from his pocket and starting to peel the notes apart. "This one's a - a - a ten? Ah yes, I see the little number on it now...So this is a five?"
"A twenty," Harry corrected him in an undertone, uncomfortably aware of Mr. Roberts trying to catch every word.
"Ah yes, so it is....I don't know, these little bits of paper..."
"You foreign?" said Mr. Roberts as Mr. Weasley returned with the correct notes.
"Foreign?" repeated Mr. Weasley, puzzled.
"You're not the first one who's had trouble with money," said Mr. Roberts, scrutinizing Mr. Weasley closely. "I had two try and pay me with great gold coins the size of hubcaps ten minutes ago."
"Did you really?" said Mr. Weasley nervously.
Mr. Roberts rummaged around in a tin for some change.
"Never been this crowded," he said suddenly, looking out over the misty field again. "Hundreds of pre-bookings. People usually just turn up...."
"Is that right?" said Mr. Weasley, his hand held out for his change, but Mr. Roberts didn't give it to him.
"Aye," he said thoughtfully. "People from all over. Loads of foreigners. And not just foreigners. Weirdos, you know? There's a bloke walking 'round in a kilt and a poncho."
— Harry Potter and the Goblet of Fire, Chapter 6: The Portkey
If you compare typical Muggles with wizards, you will get a huge chance to distinguish wizards as odd people. The ways they talk (e.g. "Hold your Hippogriffs") and look (especially when trying to look like Muggles) makes them distinguishable from Muggles. You can always ask person what time is it or to change some money? They probably will have some troubles with this simple activities.
If a wizard is really good at pretending to be a Muggle, you can use Legilimency and see through them. Muggles will have no power to hide their thoughts. Only masters of Occlumency will be able to give you false thoughts. So if someone blocks his mind he probably is a wizard.
"The mind is not a book, to be opened at will and examined at leisure. Thoughts are not etched on the inside of skulls, to be perused by any invader. The mind is a complex and many-layered thing, Potter. Or at least most minds are... It is true, however, that those who have mastered Legilimency are able, under certain conditions, to delve into the minds of their victims and to interpret their findings correctly."
— Harry Potter and the Order of the Phoenix, Chapter 24: Occlumency