This is the scene in question (emphasis mine):
With a sigh, he seated himself on the edge of his bed. The lamps were already lit, more than a dozen, leaving no shadow anywhere. The tent was as bright inside as noonday. “Have you thought over my proposal? Accept, and you walk free. Refuse, I know how to hurt your sort. I can make you scream through endless dying. Forever dying, forever screaming.”
The Wheel of Time, The Great Hunt
In The Eye of the World we find out that he has somehow "merged" with Mordeth. That, in addition to being the Dark One's Hound, gives him his unspecified powers.
Other than my emphasis, there really is no other reference of how exactly Fain manages to subdue a Myrddraal.
Supported by this interview (Q4)
Q: In The Great Hunt, who wrote the Dark Prophecy on the dungeon wall in Fal Dara? And why, after Ingtar released Padan Fain from the dungeon, did Fain decide to go to Toman Head? We know he was rebelling against Ishamael's orders (he was supposed to follow the Myrddraal to Shayol Ghul) but why did Fain go to Cairhien and then to Toman Head?
A: Fain (now amalgamated with Mordeth) was seeking his own power base, something he would try again with Pedron Niall and Toram Riatin. He wanted enough power to be able to kill Rand, Mat and Perrin, though most especially Rand, and to protect himself against agents of the Shadow. Because of Darkfriend reports, the Myrddraal who wrote the prophecy already knew who the strangers on Toman Head were, or claimed to be: Artur Hawkwing's armies returned to reclaim the lands stolen from Hawkwing's heirs. He knew that they collared women who could channel, which appealed to Fain/Mordeth, since one disliked Aes Sedai at best and the other purely hated them. The Myrddraal didn't simply give this up to Fain, you understand. Fain is one of the few people who could successfully torture information out of one of the Eyeless. As for why he went to Cairhien first, he knew the location of the Waygate there (along with several others and how to read the guidings in the Ways, this last from Mordeth) and preferred to use the Ways rather than make the longer cross-country journey from Fal Dara to Toman Head.
I don't believe RJ/BS have ever fully explained Fain's special powers. However, we can ascertain that he exerts some kind of fear over the other Shadowspawn, maybe because of his nature, his merging with Mordeth and the dagger. We don't really know.
Ordeith smiled. It thought him a fool. It would learn. They all would. “First, the matter of, shall we say, agreements and accord?” As he talked, the Myrddraal began to sweat.
I read this as Fain being able to induce fear into the Myrddraal, (probably) the same way a Myrddraal induces fear when it looks at people. Then again, it's all speculative. But we can tell for sure that his special powers, at least, makes him able to subdue Myrddraal.