Under The Garden by Graham Greene.
Yes, this is a very compelling story, that I too sometimes think about decades after first reading it. Hard to even say what it was about, except it sticks with you. Luckily, in high school in the 90s I started a personal collection of quotes I liked, shortly after I read this, and have it listed in that collection.
The legless man and his tongueless wife lived underground based on a theory that only accidents killed you, by living in a safe place one could be effectively immortal. The narrator remembers this as an adult, when he is diagnosed with cancer, and goes looking for the half remembered tunnel to the underground as a chance to be saved.
The legless man dispensed an array of interesting wisdom, such as, "Be disloyal. It's your duty to the human race. The human race needs to survive and it's the loyal man who dies first from anxiety or bullet or overwork."
Excerpt from a review:
These chapters remind me strongly of John Cowper Powys as we follow WW (one moit (a word used in the text) himself as a child dreaming and the other moit himself now older, in terminal illness, remembering real events as a child under the garden), a mix of Long John Silver and Robinson Crusoe, and finding first this fish and a newspaper (Colchester Guardian) from fifty years before smelling of fish, and an oldish man sitting on a decorative lavatory seat (except there is a pit underneath) and his quacking wife.