I'm hazy on the details, and I've tried finding this story earlier to no avail, but as far as I can recall:
An entrepreneur(?) raises a team of mercenaries with the intent to ascend an alien artefact/tower located on a barren world. The tower proves impenetrable to external access save through the front entrance. Each room of the tower is one in of a series of progressively difficult mathematical problems. Access to the subsequent room is gained by solving the problem, whereas incorrect answers result in increasingly severe punishments, all grisly and unpredictable. It is hypothesized that surmounting the tower will lead to first-contact with an alien race existing on a plane beyond traditional intelligence, though later in the story it is intimated that the tower is a cosmic joke, a never-ending sequence of deadly rooms.
As far as I remember, the galaxy is structured in a free-form trade alliance of alien cultures derived from human stock, from which the team members are drawn:
- An ethereal, spiritual word of telepaths. I recall visualizing the planet as consisting of filamentous, gossamer, orange/pink glowing structures. The culture seems to have a rare yet perennial festival involving the linking of all minds, and the protagonist is visiting this world with the intent of participating when he? is recruited.
- An industrialized cybernetic world of both physical and mental augmentations/implants.
The team is pushed beyond all reasonable limits before
retreating, suffering the loss of several members. If I recall correctly, the story concludes with the protagonist developing an obsession over mounting the tower (despite the apparent pointlessness), returns to planet  to replace parts lost in the first attempt as well as gain enhanced cybernetic mathematical capacities before intending to ascend the tower solo.
It was part of a massive anthology of short science-fiction stories, another of which I would like to be able to recall. If I can't find the anthology from the answer, I'll probably post another question.