The Witch-King was certainly aware of Glorfindel's prophecy (it's described as being remembered by many people) and he seems to have had no fear in explaining to his enemies that he was, to his own mind, essentially immortal because of it.
During his fight with Éowyn, he explicitly references the wording of the prophecy. e.g. that he cannot fall "by the hand of man":
"A sword rang as it was drawn. “Do what you will; but I will hinder
it, if I may.”
“Hinder me? Thou fool. No living man may hinder me!”
When it's then revealed that his opponent isn't in fact a man (but rather a woman) he seems to hesitate for a few seconds, presumably contemplating whether this makes a substantial difference.
“But no living man am I! You look upon a woman. Éowyn I am, Éomund's
daughter. You stand between me and my lord and kin. Begone, if you be
not deathless! For living or dark undead, I will smite you, if you
touch him.”
The winged creature screamed at her, but the Ringwraith made no
answer, and was silent, as if in sudden doubt.
As to the second part of your question (e.g. why he keeps fighting after it's revealed that she's a woman), it seems to boil down to the fact that running away from someone a fraction of his size would be ridiculous. On top of that, Éowyn kills his mount so it's not like he's going anywhere in a hurry.
Still she did not blench: maiden of the Rohirrim, child of kings,
slender but as a steel-blade, fair but terrible. A swift stroke she
dealt, skilled and deadly. The outstretched neck she clove asunder
Backward she sprang as the huge shape crashed to ruin, vast wings
outspread, crumpled on the earth; and with its fall the shadow passed
away and the hewn head fell like a stone.
For the record, he was actually doing quite well against her until he got stabbed from behind (by one of those sneaky little hobbitses):
Merry's sword had stabbed him from behind, shearing through the black
mantle, and passing up beneath the hauberk had pierced the sinew
behind his mighty knee.
“Éowyn! Éowyn!” cried Merry. Then tottering, struggling up, with her
last strength she drove her sword between crown and mantle, as the
great shoulders bowed before her.
The order of events in the film version is similar, except that she kills his Nazgul mount first, then he crows at her (without her part of the dialogue). He also shows none of the hesitation suggested in the novel.