This sounds like Invitation to Hell. The helmet was a component of a spacesuit being developed by the company for which the protagonist was working.
In the first test of the helmet, it was aimed at a human and a houseplant, and it analyzed them as "human - terrestrial - benign" and "non-human - terrestrial - benign" respectively.
The scientist for whom the helmet produced an analysis of "non-human - malignant" was a member of the local country club that was pressuring the protagonist and his family to join.
Other scenes that I remember:
- The protagonist wearing the spacesuit at a function hosted by the country club - all of the country club members are analyzed as "non-human - malignant".
- The protagonist's wife becoming violent, leading the protagonist to suspect that she has been kidnapped and replaced.
- The protagonist going through a door at the country club and finding the firey caverns of Hell on the other side.
- The protagonist finding his real wife in Hell, where she was under a compulsion to play a piano.