Novel where character travels through hell and loses their memory

Question

The character is traveling though a stereotypical hell (rocky, barren, hot, occasional river of lava) and they are smart enough to stay away from being captured by demons, but there are other people locked in cages etc.

I think I remember one scene where some demons are in some sort of a building with a waterwheel made of bones and, but the river is made of lava (or blood?) instead of water.

They spend so long subjectively in hell that they lose their memory and forgot who they are.

The twist (I think) is that hell is just a simulation, and the character manages to leave, but then for some reason has to decide to return.

Answer 1

This sounds to me like Surface Detail by Iain M. Banks. One of the subplots involves a society whose government runs a Hell simulation.

Prin and Chay belong to a species that use the threat of Hell to control the behaviour of their population. While still alive, they enter the Pavulean Hell on a mission to reveal the existence and details of this Hell to the general population. Prin succeeds in getting out, but has to leave his partner Chay behind, where she is subsequently tortured, restored to some semblance of sanity, and finally given a role as an angel in Hell who is able to release one soul a day by killing them. Prin testifies to the Pavulean parliament of his experience in Hell and attempts to convince them that it should be abolished.