I read a story or a book by Neil Gaiman that I believe was set in Norse mythology but I don't think was his actual book "Norse Mythology," although I could be wrong about this. Whatever it was, he referenced a set of caves that lead to every place on Earth.
A partial answer might be "The Truth is a Black Cave in the Mountains" as it involves someone entering a set of caves in search of a treasure associated with Norse mythology. That said, I'm failing to find any reference to the caves leading to every place on Earth.
Summary from this article in Exuent magazine:
The Truth is a Cave in the Black Mountains is a long short story or a short novella (and apparently these can be referred to as novelettes). In any case, read beautifully, with a few pauses for music, it takes a little over an hour, and the number of plot beats feels very similar to a feature film, and as satisfying. It is a tale set in Scotland long ago, a world of clans, dirks, tablet, porridge and misty isles. Our narrator, a wry ‘wee man’ with incredible strength and agility, employs a gruff Reaver to guide him to a mythical cave in the mountains on an island which is sometimes absent, from which any man may carry as much gold as he wishes. But there’s a price. There’s always a price.