This is a question of some controversy among Tolkien fans.
There probably is a physical component of the Eye; though we don't know what form that takes (whether it is a red flaming eye, or whether it's just Sauron magically enhancing his own vision), there are references that are hard to reconcile otherwise.
However, I would argue that most references to it are not referring to a literal Eye on the top of a tower, but rather to one of two more abstract concepts:
- The focus of Sauron's attention
- Sauron's malicious will, hunting for the Ring
The Physical Component
In Return of the King, we have the following passage (emphasis mine):
[R]ising black, blacker and darker than the vast shades amid which it stood, the cruel pinnacles and iron crown of the topmost tower of Barad-dûr. One moment only it stared out, but as from some great window immeasurably high there stabbed northward a flame of red, the flicker of a piercing Eye; and then the shadows were furled again and the terrible vision was removed. The Eye was not turned to them: it was gazing north to where the Captains of the West stood at bay
Return of the King Book VI Chapter 3: "Mount Doom"
Shortly before this we're told of a "Window of the Eye" in Barad-dûr. I'm at a loss to explain either of these passages without the Eye being at least partially physical.
One of the main argument brought forth for the Eye being physical, however, I do disagree with; the passage is from Frodo's vision in the Mirror of Galadriel:
[S]uddenly the Mirror went altogether dark, as dark as if a hole had opened in the world of sight, and Frodo looked into emptiness. In the black abyss there appeared a single Eye that slowly grew. until it filled nearly all the Mirror. So terrible was it that Frodo stood rooted, unable to cry out or to withdraw his gaze. The Eye was rimmed with fire, but was itself glazed, yellow as a cat's, watchful and intent, and the black slit of its pupil opened on a pit, a window into nothing.
Fellowship of the Ring Book II Chapter 7: "The Mirror of Galadriel"
I've questioned this interpretation elsewhere on the site, and I stand by that argument.
The Metaphorical Component
As I said, there seem to be, broadly, two metaphors involving the Eye. The simplest one is simply Sauron's attention, the same way "eye" is often used idiomatically in English. Consider the following passages:
I Grishnákh say this: Saruman is a fool. and a dirty treacherous fool. But the Great Eye is on him.
The Two Towers Book III Chapter 3: "The Uruk-Hai"
There was some link between Isengard and Mordor, which I have not yet fathomed. How they exchanged news I am not sure; but they did so. The Eye of Barad-dûr will be looking impatiently towards the Wizard's Vale, I think; and towards Rohan. The less it sees the better.
The Two Towers Book III Chapter 10: "The Voice of Saruman"
[T]hey couldn't get Lugbúrz to pay attention for a good while, I'm told.'
'The Eye was busy elsewhere, I suppose,' said Shagrat. `Big things going on away west, they say.'
The Two Towers Book IV Chapter 10: "The Choices of Master Samwise"
It was then ten days since the Ring-bearer went east from Rauros, and the Eye of Sauron, I thought, should be drawn out from his own land.
Return of the King Book V Chapter 9: "The Last Debate"
The more interesting case is the use of the Eye as a metaphor for Sauron's hunt for the Ring; I've argued elsewhere that Frodo's vision in the Mirror of Galadriel is this, but this seems to be what is generally meant by "the Eye" when it's referred to in Frodo's context; the first time we see this iconography, though, is actually from Bilbo:
[The Ring] has been so growing on my mind lately. Sometimes I have felt it was like an eye looking at me.
Fellowship of the Ring Book I Chapter 1: "A Long-expected Party"
This is unquestionably a metaphor, and bears some similarity to some of Frodo's later descriptions of the Eye as a malicious presence, that seems to be watching him; for example:
[F]ar more he was troubled by the Eye: so he called it to himself. It was that more than the drag of the Ring that made him cower and stoop as he walked. The Eye: that horrible growing sense of a hostile will that strove with great power to pierce all shadows of cloud, and earth, and flesh, and to see you: to pin you under its deadly gaze, naked, immovable.
The Two Towers Book IV Chapter 2: "The Passage of the Marshes"
Despite what is claimed in the question, this seems pretty unambiguously metaphorical to me; note that the Eye is "what he called it to himself," where "it" is that hostile will (of Sauron's, clearly). I also find this description quite similar to the one given when Frodo puts on the Ring at Amon Hen:
And suddenly he felt the Eye. There was an eye in the Dark Tower that did not sleep. He knew that it had become aware of his gaze. A fierce eager will was there. It leaped towards him; almost like a finger he felt it, searching for him. Very soon it would nail him down, know just exactly where he was. Amon Lhaw it touched. It glanced upon Tol Brandir he threw himself from the seat, crouching, covering his head with his grey hood.
Fellowship of the Ring Book II Chapter 10: "The Breaking of the Fellowship"
I find it notable here that Frodo doesn't see an Eye; he feels it. I seems to me that this is also a metaphorical description; Frodo is experiencing some manner of malicious, hunting will, and he describes that using the image of an Eye.