I am not sure if there is a real-life equivalent but it seem that the "finger dance" is just throwing an axe at your friend/sparring partner.
Three burly men were doing the finger dance, spinning short-hafted axes at each other. The trick was to catch the axe or leap over it without missing a step. It was called the finger dance because it usually ended when one of the dancers lost one . . . or two, or five.
A Clash of Kings - Theon II
It is never fully explain why the Iron Islanders practice this game; although one would assume it is to improve your dexterity in battle, but it probably arose out of boredom (the Iron Island are often described as a bleak and boring place). As such it doesn't seem that participants are upset with a competitor when the dance ends in loss of limb.
It had been his axe that sheared off Urri's hand, whilst they danced the finger dance together, as friends and brothers will.
A Feast for Crows - The Prophet
Qarl the Maid and Eldred Codd danced the finger dance. A roar of laughter went up when one of Eldred's fingers landed in Ralf the Limper's wine cup.
A Feast for Crows - The Iron Captain
It even seems that the dance can be used to settle large matters, even who should be King of the Iron Islands (according to legend anyway).
Afterward the victors could not agree on who should succeed Rognar as king, so it was decided that they would settle the matter by dancing the finger dance, a game popular amongst the ironborn wherein players spin a throwing axe at one another and attempt to snatch it from the air. Harras Hoare emerged as victor, at the cost of two fingers. As Harras Stump-hand, he ruled the Iron Islands for thirty years.
The World of Ice and Fire - The Iron Islands: The Iron Kings
This is also the best description we have of the finger dance... The descriptors "spinning" and the tactic of "leaping over it" seem to suggest that the axe is thrown at more horizontal angle, but that is just a hunch of mine.