In The Facts of M. Valdemar's Case, an 1845 short story by Edgar Allen Poe, a mesmerist attempts to hypnotize a man at the moment of his death. Poe pays attention to the eyes at different points of the story, in a way that is mostly physiologically accurate, and eventually reaching a more horrific level. This is not exactly the same as what you're asking about, but I think it's similar in that the cartoonish or fantastic effects on the eyes, in more recent works, are really an exaggerated version of what happens in a trance state.
As the hypnosis begins:
The glassy roll of the eye was exchanged for that expression of uneasy inward examination which is never seen except in cases of sleep-waking, and which it is quite impossible to mistake.
Later:
the eye-lids unclosed themselves so far as to display a white line of the ball
At the moment of death:
The eyes rolled themselves slowly open, the pupils disappearing upwardly
His body then seems to be in a totally paused state. Seven months later, they decide to revive him from the trance:
The first indication of revival was afforded by a partial descent of the iris. It was observed, as especially remarkable, that this lowering of the pupil was accompanied by the profuse out-flowing of a yellowish ichor (from beneath the lids) of a pungent and highly offensive odor.
There are other works where people are noted to have subtle differences in their eyes when hypnotized or otherwise suborned -
- In Daphne du Maurier's Trilby (1894), the title character is hypnotized by Svengali to become a talented singer. An observer notes: "Her face was narrower and longer, her eyes larger, and their expression not the same; then she seemed taller and stouter, and her shoulders broader and more drooping, and so forth."
- In C. S. Lewis's The Lion, the Witch, and the Wardrobe (1950), a character under the Witch's spell is described with: "You can always tell them if you've lived long in Narnia, something about their eyes."
- In John Wyndham's The Midwich Cuckoos (1957), telepathic force is exerted against a man so that his "mouth went slack, his jaws fell a little, his eyes widened, and seemed to go on widening." (This is inducing fear specifically, not putting him into a trance.)
- In Madeline L'Engle's A Wrinkle in Time (1962), a character is enthralled by the villain: "After a moment it seemed that his eyes were no longer focusing. The pupils grew smaller and smaller, as though he were looking into an intensely bright light, until they seemed to close entirely, until his eyes were nothing but an opaque blue." (This is not his eyes changing into a solid colour, but an extreme contraction of the pupils, so it's still a normal physiological effect.)
The same sort of idea seems to also apply to the eyes of the hypnotist. Both realistic and fantastical hypnotists are often described as having eyes that are somehow uniquely attractive, large, powerful, etc. That goes for Svengali, Dracula, and Dr. Mabuse. Others, like the children in The Midwich Cuckoos with their golden eyes, have an exaggerated version that does not pass as normal. These probably draw from the historical idea of the "evil eye" as well.
In film, early 20th century renditions of hypnotists such as Svengali or Mabuse use a spectrum of effects - from "intense staring eyes", to flashing effects, to distortion of the whole upper face (as in Fritz Lang's 1933 The Testament of Dr Mabuse). The victim of hypnotism having similar treatment to the eyes seems like an obvious extension, but I don't know precisely when this first appears in film. Certainly these early works have the actors of mesmerized characters variously staring, rolling their eyes, gazing vacantly, and so on. It may be hard to distinguish whether a particular rendition is meant to be supernaturally heightened, or not, given the visual style of cinema at the time.