Oaths are words, Words are wind
Catelyn Stark answered your question when Robb wondered the same. If Freys are bannermen to Tullys, they ought to let Starks pass.
Walder Frey, Lord of the Crossing, had assembled a force of near four
thousand men at his castles on the Green Fork.
“Late again,” Catelyn murmured when she heard. It was the Trident all
over, damn the man. Her brother Edmure had called the banners; by
rights, Lord Frey should have gone to join the Tully host at Riverrun,
yet here he sat.
“Four thousand men,” Robb repeated, more perplexed than angry. “Lord
Frey cannot hope to fight the Lannisters by himself. Surely he means
to join his power to ours.”
“Does he?” Catelyn asked. She had ridden forward to join Robb and
Robett Glover, his companion of the day. The vanguard spread out
behind them, a slow-moving forest of lances and banners and spears. “I
wonder. Expect nothing of Walder Frey, and you will never be
"He's your father's bannerman."
"Some men take their oaths more seriously than others, Robb. And Lord
Walder was always friendlier with Casterly Rock than my father would
have liked. One of his sons is wed to Tywin Lannister's sister. That
means little of itself, to be sure. Lord Walder has sired a great many
children over the years, and they must needs marry someone. Still
AGOT - Catelyn IX
Inside the dark corner called Walder's mind
In the same chapter, she also explained how Lord Walder's mind works:
"Do you think he means to betray us to the Lannisters, my lady?"
Robett Glover asked gravely.
Catelyn sighed. "If truth be told, I doubt even Lord Frey knows what
Lord Frey intends to do. He has an old man's caution and a young man's
ambition, and has never lacked for cunning."
Precedent for vassals turning against Overlords in favor of Crown
Earlier, Catelyn wondered where the loyalties of her Father's bannermen lie:
Her father was the staunchest man who'd ever lived, and she had no
doubt that he would call his banners … but would the banners come? The
Darrys and Rygers and Mootons had sworn oaths to Riverrun as well, yet
they had fought with Rhaegar Targaryen on the Trident, while Lord Frey
had arrived with his levies well after the battle was over, leaving
some doubt as to which army he had planned to join (theirs, he had
assured the victors solemnly in the aftermath, but ever after her
father had called him the Late Lord Frey).
AGOT - Catelyn
Lord Walder's Position
Lord Frey however said that he wanted to go to Riverrun but he was waiting for his full levies to gather and Edmure lost before that. As to his vows, he correctly points out that he swore vows to the Crown as well.
“Now that I have observed the courtesies, my lady, perhaps my sons
will do me the honor of shutting their mouths. Why are you here?”
“To ask you to open your gates, my lord,” Catelyn replied politely.
“My son and his lords bannermen are most anxious to cross the river
and be on their way.”
“To Riverrun?” He sniggered. “Oh, no need to tell me, no need. I’m not
blind yet. The old man can still read a map.”
“To Riverrun,” Catelyn confirmed. She saw no reason to deny it. “Where
I might have expected to find you, my lord. You are still my father’s
bannerman, are you not?”
“Heh, “ said Lord Walder, a noise halfway between a laugh and a grunt.
“I called my swords, yes I did, here they are, you saw them on the
walls. It was my intent to march as soon as all my strength was
assembled. Well, to send my sons. I am well past marching myself, Lady
Catelyn.” He looked around for likely confirmation and pointed to a
tall, stooped man of fifty years. “Tell her, Jared. Tell her that was
“It was, my lady,” said Ser Jared Frey, one of his sons by his second
wife. “On my honor.”
“Is it my fault that your fool brother lost his battle before we could
march?” He leaned back against his cushions and scowled at her, as if
challenging her to dispute his version of events. “I am told the
Kingslayer went through him like an axe through ripe cheese. Why
should my boys hurry south to die? All those who did go south are
running north again.”
AGOT - Catelyn IX
“We want to cross,” Catelyn told him.
“Oh, do you? That’s blunt. Why should I let you?”
For a moment her anger flared. “If you were strong enough to climb
your own battlements, Lord Frey, you would see that my son has twenty thousand men outside
“They’ll be twenty thousand fresh corpses when Lord Tywin gets here,”
the old man shot back. “Don’t you try and frighten me, my lady. Your
husband’s in some traitor’s cell under the Red Keep, your father’s
sick, might be dying, and Jaime Lannister’s got your brother in
chains. What do you have that I should fear? That son of yours? I’ll
match you son for son, and I’ll still have eighteen when yours are all
“You swore an oath to my father,” Catelyn reminded him.
He bobbed his head side to side, smiling. “Oh, yes, I said some words,
but I swore oaths to the crown too, it seems to me. Joffrey’s the king
now, and that makes you and your boy and all those fools out there no
better than rebels. If I had the sense the gods gave a fish, I’d help
the Lannisters boil you all.”
AGOT - Catelyn IX
Lord Walder Remembers
Then he expressed his resentment on various insults by House Tully:
Your lord father did not come to the wedding. An insult, as I see it.
Even if he is dying. He never came to my last wedding either. He calls
me the Late Lord Frey, you know. Does he think I’m dead? I’m not dead,
and I promise you, I’ll outlive him as I outlived his father. Your
family has always pissed on me, don’t deny it, don’t lie, you know
it’s true. Years ago, I went to your father and suggested a match
between his son and my daughter. Why not? I had a daughter in mind,
sweet girl, only a few years older than Edmure, but if your brother
didn’t warm to her, I had others he might have had, young ones, old
ones, virgins, widows, whatever he wanted. No, Lord Hoster would not
hear of it. Sweet words he gave me, excuses, but what I wanted was to
get rid of a daughter.
“And your sister, that one, she’s full as bad.
I proposed that Lord and Lady Arryn foster two of my grandsons at
court, and offered to take their own son to ward here at the Twins.
Are my grandsons unworthy to be seen at the king’s court?
Lord Arryn wouldn’t have him, or the other one, and I blame your lady
sister for that. She frosted up as if I’d suggested selling her boy to
a mummer’s show or making a eunuch out of him
AGOT - Catelyn IX
So in conclusion:
- Lord Walder is a cautious man who takes no unnecessary risks. Why should he take field for a side that was losing?
- Lord Walder is legally bound to follow his liege lord but he is also legally bound to be faithful to the Crown. Of course vows mean little to a man like him, he has never been overly conscious about them. But there is precedent, in previous wars against the Crown, many Lords forswore their local overlords and fought for the King.
- He has Kinship with Lord Tywin. Lord Tywin's beloved sister Genna Lannister is his daughter-in-law. He can be sure that even if Tywin Lannister destroys everyone else, House Frey will be secure, leaving him little to worry about.
- Walder Frey could only sense ruin in Tully camp, why should he join it? Any Tully Power which could exact retribution is shattered by Jaime. Of course he is not blind, the Starks were here and sooner or later it was expected Arryns would be here as well. Together, they gave the Tullys a winning chance but to enter Riverlands without risking a battle with Tywin Lannister at Ruby Ford, they needed Lord of the Crossing which gave him enormous leverage, something he wasn't willing to give up for nothing. Now was the time to get something in return.
- Lord Walder is not overly fond of Tullys, Starks, Arryns or Lannister. He hates that they all look down upon him. He hates that Lord Hoster Tully didn't take his daughter as daughter in law. He hates that Arryns refused to foster his grand kids.
So TLDR, Tullys could order the Freys but would the Freys listen? Unlikely.