Firstly, I'd like to catalog the major mentions of the old blood and see what they have to say about what the old blood is supposed to do:
Powers/Abilities corresponding to The Old Blood
- Purity/strength of the old blood is clearly tied to the ability to use The Power:
"Do you have any idea," Alanna went on, "how many towns and villages we usually must visit to find three girls with the ability inborn? The only wonder is that it took us so long to come hunting more. The old blood is very strong here in the Two Rivers."
The Shadow Rising, Chapter 31.
Additionally, it's clear that in-universe, there is a means of accessing ancestral memories distinct from the Aelfinn/Eelfinn's powers, as the ter'angreal in Rhuidean shows.
While it's not definite that Mat is using the memory of an ancestor, it is clear that the Aes Sedai believe that Mat's strength in the old blood is the cause of his random fits of using the Old Tongue, as the above quote and the following show:
[as Mat is being healed and separated from the Shadar Logoth dagger, he screams stuff in the Old Tongue] "Fascinating," Verin said. "That the Old Blood could flow so strongly in anyone today."
[. . .]
"What was that he was shouting, Mother?" Elayne asked, then hastily added, "If I may ask."
"He was ordering soldiers." The Amyrlin gave the young man lying on the table a quizzical look. [...] "In a battle two thousand years gone, I would say. The Old Blood comes again."
[. . .]
"For a time," she said firmly, "I believe the past and the present were one. He was there, and he was here, and he knew who we were. He commanded us to release him."
The Dragon Reborn, Chapter 18
Now, Aes Sedai cannot lie, and while they never directly say "this is because of the old blood" -- they only say "Mat speaks in the Old Tongue sometimes. The Old Blood is strong in The Two Rivers" -- we don't have any real reason to discredit it since they even use it as justification to send two Aes Sedai to a random backwater province of Andor.
It's also not totally unheard-of in-universe by non-Aes Sedai, as Thom indicates (and Agelmar later corroborates):
"The old blood, she said. The blood, not a dead man. I've heard that it can happen, sometimes. Heard, though I never really thought. . . . It was your roots, boy. A line running from you to your father to your grandfather, right on back to Manetheren, and maybe beyond. Well, now you know your family is old."
The Eye of the World, Chapter 19
And the old blood is justification enough for Agelmar to let the "Farmboys" go into the Blight, without additional troops, to seek the Eye of the World, more so than Moiraine's claim that they are ta'veren!
"They are ta'veren," Moiraine said soothingly. [explains three ta'veren together can change the course of history]
Agelmar stopped trying to find his sword, but he still looked at Rand and the others doubtfully. "Moiraine Sedai, if you say they are, then they are, but I cannot see it. Farmboys. Are you certain, Aes Sedai?"
"The old blood," Moiraine said, "split out like a river breaking into a thousand times a thousand streams, but sometimes streams join together to make a river again. The old blood of Manetheren is strong and pure in almost all these young men. Can you doubt the strength of Manetheren's blood, Lord Agelmar?"
[. . .]
"Manetheren," Agelmar said slowly, nodding. "I would not doubt that blood. [. . .] It shall be done as you wish, Aes Sedai."
The Eye of the World, Chapter 47
Although, this passage could be just Agelmar believing more in ancestral martial prowess than in the mysterious workings of the Wheel.
However, Birgitte has perhaps the clearest explanation of manifestations of the Old Blood, driven by her lifetimes of experience, as Mat tries to explain his ability to speak the Old Tongue:
"An Aes Sedai once told me the old blood runs strong in -- What are you bloody well laughing at now?"
"You, Mat. [. . .] Some people speak a few words, a phrase or two, because of the old blood. Usually what they say, or not quite. But you. . . . One sentence you're an Eharoni High Prince and the next a First Lord of Manetheren, accent and idiom perfect."
A Crown of Swords, Chapter 21
Hence, I conclude that we must accept that the Old Blood is the cause of his memories prior to the Finn, and the reason it only happens to him is that Mat just has the strongest bloodline ties to Manetheren out of the group.
As the above quotes show, Mat can clearly occasionally live out the memories of men long dead, quite apart from having the gaps filled by the Finn, indicating his long ancestry. Mat is not the only one in whom the old blood runs strongly, though, as Egwene almost understands his warcry:
Mat shrugged uncomfortably. "I don't remember." He stared at them deffensively. "Well, I don't. It's all foggy. I don't know what it was, or where it came from, or what it means." He gave a self-deprecating laugh. "I don't suppose it means anything."
"I . . . I think it does," Egwene said slowly. "When you shouted, I thought--just for a minute--I thought I understood you. But it's all gone, now."
[. . .]
Rand thought he knew what Mat was thinking. The same thing he was thinking. If Mat was a descendant of the ancient kings of Manetheren, maybe the Trollocs were really after him and not all three of them. The thought made him ashamed. His cheeks colored, and when he caught a guilty grimace on Perrin's face, he knew Perrin had been having the same thought.
The Eye of the World, Chapter 18.
As an aside, Mat has the following to say upon finding out about the old blood:
"It's as if some dead man was speaking with my mouth. I don't like it." The Eye of the World, Chapter 19.
Which is ironic given his future(?) memories of past lives as well as a rather amusing explanation for his reluctance to be tied to the Horn (and probably being reborn but perhaps he is only ever reborn in the Third Age).
