I remember reading this story about 20 years ago as part of a scifi/fantasy compilation.
It's about a guy remembering a very weird event that happened when he was a child, and many years later returning to the place where it happened. He remembers a hole that he fell through, and this scary old man with no legs that lived down there. The legless man had a mute wife, that "quacked", and there were golden trinkets in his lair. I think the old man wanted the boy to stay down in the hole, and while the boy pretended to accept his new fate, was always looking for a way to escape. In the end, the boy tries to get out and the legless man tries to catch him, dragging himself through the tunnels with his powerful arms.