I think there’s a certain amount of context. Or as Obi Wan would say: “From a certain point of view.”
The Paz Vizsla (aka: Heavy Infantry Mandalorian) is questioning the protagonist’s (Din Djarin) courage:
“Our world was shattered by the Empire with whom this coward shares tables.”
Then he tries to remove the protagonist’s helmet, which triggers the fight. Disrespect was meant and given out by that action. The helmet is clearly important and removal of it was meant to be symbolic.
“When one chooses to walk the way of the Mandalore, you are both hunter and prey.”
There’s a certain reputation to that armour, and considering the Empire apparently has them in hiding, even after they are gone, there’s obviously a certain risk to being seen as a Mandalorian.
“How can one be a coward if one chooses this way of life?”
Kinda backs it up. There’s a risk to being seen as a Mandalorian.
“Have you ever removed your helmet?”
In previous context — is almost like saying, “Have you ever denied being a Mandalorian?”
Considering Paz’s distaste for hiding this makes sense:
“Has it ever been removed by others?”
Would presumably suggest defeat.
I suspect this doesn’t mean they live in their helmets — and considering the response “This is the way…” and how that immediately stopped the fight — I suspect there was some cultural and religious implications there.
I’m also pretty sure the protagonist will lose his helmet at some point…
While I’ve yet to watch the episode in question, this recap on io9 seems to confirm that the protagonist does remove his helmet to eat:
“Mando hasn’t taken off his helmet in front of people since he was a kid, but (as many people wondered after the comments last week) he does take it off to eat if he’s alone.”