Lord Petyr Baelish tells Catelyn Stark that the Valyrian steel dagger is Tyrion's, thus getting him arrested. I don't think Tyrion Lannister would have really appreciated his little journey to the Vale. So, why did he do nothing against Littlefinger once he returned to King's Landing?
Short answer : he wanted to but he couldn't
Long answer :
Petyr Baelish was quite powerful in King's Landing, and once Tyrion got more influence, Littlefinger became Lord of Harrenhal and went to the Vale, so too powerful and too far from the influence of Tyrion.
He really didn't appreciate his journey in the Vale, but didn't know how to confront Littlefinger :
"Indeed." Jowly, balding Janos Slynt looked rather like a frog, a smug frog who had gotten rather above himself. "We have sore need of you, my lord. Rebellion everywhere, this grim omen in the sky, rioting in the city streets . . ."
"And whose fault is that, Lord Janos?" Cersei lashed out. "Your gold cloaks are charged with keeping order. As to you, Tyrion, you could better serve us on the field of battle."
He laughed. "No, I'm done with fields of battle, thank you. I sit a chair better than a horse, and I'd sooner hold a wine goblet than a battle-axe. All that about the thunder of the drums, sunlight flashing on armor, magnificent destriers snorting and prancing? Well, the drums gave me headaches, the sunlight flashing on my armor cooked me up like a harvest day goose, and those magnificent destriers shit everywhere. Not that I am complaining. Compared to the hospitality I enjoyed in the Vale of Arryn, drums, horseshit, and fly bites are my favorite things."
Littlefinger laughed. "Well said, Lannister. A man after my own heart."
Tyrion smiled at him, remembering a certain dagger with a dragonbone hilt and a Valyrian steel blade. We must have a talk about that, and soon. He wondered if Lord Petyr would find that subject amusing as well.
-A Clash of Kings - Tyrion I
Later, both have an interesting dialogue
"That's a handsome knife as well." "Is it?" There was mischief in Littlefinger's eyes. He drew the knife and glanced at it casually, as if he had never seen it before. "Valyrian steel, and a dragonbone hilt. A trifle plain, though. It's yours, if you would like it." "Mine?" Tyrion gave him a long look. "No. I think not. Never mine." He knows, the insolent wretch. He knows and he knows that I know, and he thinks that I cannot touch him..
-A Clash of Kings - Tyrion IV
Finally, this quote resumes Tyrion's thought about harming Littlefinger
No one had ever thought to question the appointments, and why should they? Littlefinger was no threat to anyone. A clever, smiling, genial man, everyone's friend, always able to find whatever gold the king or his Hand required, and yet of such undistinguished birth, one step up from a hedge knight, he was not a man to fear. He had no banners to call, no army of retainers, no great stronghold, no holdings to speak of, no prospects of a great marriage. But do I dare touch him? Tyrion wondered. Even if he is a traitor? He was not at all certain he could, least of all now, while the war raged. Given time, he could replace Littlefinger's men with his own in key positions, but . . .
-A Clash of Kings - Tyrion IV
Keep in mind that Tyrion was not powerful in King's Landing, and they are preparing for war against Stannis : Littlefinger even leaves King's Landing o negotiate a Lannister-Tyrell alliance, and returns only for the battle of the Blackwater. Tyrion was injured in the battle, and he takes few weeks to recover. After that, Littlefinger becomes Lord of Harrenhal, and goes to the Vale, where Tyrion could do nothing to Littlefinger.
- He understands that Littlefinger is much more powerful and dangerous than he thought
- He decides to replace Littlefinger's men with his own men, a strategy for the long term
- They don't spend a lot of time together in King's Landing
- Petyr is out of reach of Tyrion's power before Tyrion got enough power to do anything
Source: This reddit thread.
Relevant Tyrion quote from ACOK.
“That’s a handsome knife as well.”
“Is it?” There was mischief in Littlefinger’s eyes. He drew the knife and glanced at it casually, as if he had never seen it before. “Valyrian steel, and a dragonbone hilt. A trifle plain, though. It’s yours, if you would like it.”
“Mine?” Tyrion gave him a long look. “No. I think not. Never mine.” He knows, the insolent wretch. He knows and he knows that I know, and he thinks that I cannot touch him.
Another relevant quote:
“No doubt a slow one,” Tyrion said. “My lord, you were fostered at Riverrun. I’ve heard it said that you grew close to the Tullys.”
“You might say so. The girls especially.”
“I had their maidenhoods. Is that close enough?”
The lie-Tyrion was fairly certain it was a lie-was delivered with such an air of nonchalance that one could almost believe it. Could it have been Catelyn Stark who lied? About her defloration, and the dagger as well? The longer he lived, the more Tyrion realized that nothing was simple and little was true.“
To summarise, Tyrion simply isn't sure whether Catelyn was lying or Littlefinger.