Petyr Baelish is constantly referred to as "Littlefinger," and some mention that he doesn't like this name.
The first thing that comes to mind is that it's because he is not physically well-equipped, but I doubt that's the real reason.
What is the backstory behind his nickname?
The Fingers are rocky peninsulae near the Vale of Arryn, and Petyr's father was the lord of the smallest of them. And he is relatively short of stature, both as an adult and as a child when he was given the nickname.
He doesn't like this nickname as it points out he's just a minor lord and pretty much mocks him for being short sized.
Quoting directly from "A Game of Thrones" by George RR Martin;
Catelyn’s mouth grew tight. “Littlefinger,” she murmured. His face swam up before her; a boy’s face, though he was a boy no longer. His father had died several years before, so he was Lord Baelish now, yet still they called him Littlefinger. Her brother Edmure had given him that name, long ago at Riverrun. His family’s modest holdings were on the smallest of the Fingers, and Petyr had been slight and short for his age.
Moreover, answer can be found in series as well. Petyr tells it to Arya and Sansa while watching a joust (episode 4 I think) where The Mountain kills ser Hugh.
On page 167 of Game of Thrones (the first book), it says:
Her brother Edmure had given him that name, long ago at Riverrun. His family's modest holdings were on the smallest of the Fingers, and Petyr had been slight and short for his age.
Petyr Baelish is known as ‘Littlefinger’, a nickname he was given by Edmure Tully when Petyr was his ward. The name refers to the old seat of the House of Baelish, which sits on one of the smallest fingers of land that reach out into the Narrow Sea from the Vale of Arryn
Vit Michael has it right.
But, the books and the show both refer to Littlefinger's finger being little often enough, that it might be true.
Obviously, Littlefinger is going to tell his "I was short and from the Fingers" story, rather than "I'm physically lacking downstairs" (especially to a 10 year old girl).
Edmure might have just come up with a very clever nickname that mocks Baelish in more than one way.
