I'm not quite sure but it sounds like Simulacron-3 (1964), a.k.a Counterfeit World, by Daniel F. Galouye.
Simulacron 3 is the story of a virtual city (total environment
simulator) for marketing research, developed by a scientist to reduce
the need for opinion polls. The computer-generated city simulation is
so well-programmed, that, although the inhabitants have their own
consciousness, they are unaware, except for one, that they are only
electronic impulses in a computer.
The simulator's lead scientist, Hannon Fuller, dies mysteriously, and
a co-worker, Morton Lynch, vanishes. The protagonist, Douglas Hall, is
with Lynch when he vanishes, and Hall subsequently struggles to
suppress his inchoate madness. As time and events unwind, he
progressively grasps that his own world is probably not "real" and
might be only a computer-generated simulation.