Perhaps the novelette "The Six Fingers of Time" by R. A. Lafferty, first published in If, September 1960, available at the Internet Archive and Project Gutenberg and ralafferty.com. Here is part of a review from the web site Continued on Next Rock:
The odd thing in all this, of course (or an odd thing) is that the stoppage of time is likely the least original element in “Six Fingers of Time” (Robert Bloch’s “The Hell-Bound Train” is only one of many tales around that time presenting similar temporal tricks). In Lafferty’s telling, the time-manipulation and the juvenile pranks played by the protagonist are only the setup for a game played for much higher stakes than a peek or two in the women’s locker room: the ability to greatly slow time is reserved for those who have, whether vestigially or fully-formed, six fingers on each hand, and who belong to a different recension from the standard-issue man on the street.
Lafferty would use this background conspiracy more effectively in later works, especially The Devil Is Dead and the Coscuin Chronicles; here that history is compressed into the space of a novelette and made into a race against, not the time of the clock, but the span of one’s life, as the hero must learn enough (across many disciplines—yet another Lafferty autodidact) in the time he has to bring the conspiracy to light.
Thanks to its being in the public domain, “The Six Fingers of Time” is the first Lafferty story many new readers encounter, and that seems a shame. It has all the horror of a great Lafferty work, but very little of the humor; the coexistence of the two is what marks him at his peak. Yet it is undeniably a story that leaves an impression: many among the non-devoted remember it with a chill, along with its implications of a global and likely cosmic conspiracy of extradigital individuals to maintain their immortality and amusement at the expense of all others on earth. Hell, even the number six itself is charged with a certain spookiness, a “wrongness” not in the least diminished by the thought that it divides neatly into all measurements of time.
The end of the tale is ambiguous: nearing the great vision of the conspiracy’s extent, and the transmission of that vision to the public, the prematurely aged protagonist dies in his sleep, and his adversaries sedately rejoice.