The Fingers are rocky peninsulae near the Vale of Arryn, and Petyr's father was the lord of the smallest of them. And he is relatively short of stature, both as an adult and as a child when he was given the nickname.
He doesn't like this nickname as it points out he's just a minor lord and pretty much mocks him for being short sized.
Quoting directly from "A Game of Thrones" by George RR Martin;
Catelyn’s mouth grew tight. “Littlefinger,” she murmured. His face
swam up before her; a boy’s face, though he was a boy no longer. His
father had died several years before, so he was Lord Baelish now, yet
still they called him Littlefinger. Her brother Edmure had given him
that name, long ago at Riverrun. His family’s modest holdings were on
the smallest of the Fingers, and Petyr had been slight and short for