The 'Zone of Terror' refers to is a strange, unreasonable panic that the character feels when he's ascending on the conveyor belt. It's described earlier in the book in more detail:
Even under power the ascent seemed light-years long, an infinite regression of rungs and rings and bulkheads that almost seemed to breathe when he wasn’t looking. The belt drew him up through a series of telescoping segments; hazard striping highlighted the spots where each handed off to the next, where the bore of the tunnel increased by some fractional increment. Little readouts, logarithmically spaced along the bulkhead, pegged the gravity—0.3, 0.25, 0.2—as he rose.
Halfway up, the panic returned.
He had a few seconds’ warning: a sudden formless disquiet spreading through the gut, an anxiety that his civilized neocortex tried to write off as simple acrophobia. In the next instant it metastasized into a bone-chilling terror that froze him solid. Suddenly his breathing was fast as a hummingbird’s heartbeat; suddenly his fingers were clenched tight as old roots around a rock.
He waited, paralyzed, for some nameless horror to rise in his sight and tear him limb from limb. Nothing did. He forced himself to move. His head turned like a rusted valve, creaked left, right; his eyes rolled frantically in search of threats.
Nothing. An intersegmental gasket passed around him. The rungs of the ladder ticked unremarkably by. Something flickered at the corner of—but no. Nothing there.
Nothing at all.
He looks around and can't find anything to cause it so sort of writes it off as general formless anxiety probably partly spiked by fears of Valerie, a vampire who's already freaked him out and often stays out of sight.
Much later, we learn that
Valerie is the cause, but in a more direct way:
“You ever been scared roach?”
All the time. “Rakshi, we almost died—”
“Before that.” Sengupta head jerked back, forth. “Scared for no reason scared just going to the bathroom.”
Something jumped in his stomach. “What did you find?”
She threw a camera feed onto the wall: an eye in the attic, looking down along the empty compartment to the Hub hatch. Sengupta zoomed obliquely on a patch of bulkhead beside the secondary airlock. Someone had scrawled some kind of glyph across that surface, a tangle of multicolored curves and corners that might have passed for some Cubist’s rendition of a very simple neural circuit.
“I don’t remember seeing that before,” Brüks murmured.
“Yah you do you just don’t remember it. Only lasts two hundred milliseconds pure luck this showed up on a screen grab. You see it but you don’t remember it and it scares the shit out of you.”
“Not scaring me now.”
“This is just one frame roach it’s part of an animation but the cameras don’t scan fast enough and they’re all gone now. I had to sieve like a bugger to even get this much.”
He stared at the image: a jagged little tangle of lines and arabesques, a piece of abstract graffiti maybe a hand’s-width across. It almost looked meaningful when spied from the corner of the eye, like a collection of letters on the verge of forming a word; it dissolved into gibberish when you looked at it. Even cut out of sequence, even spied from this oblique angle, it made his brain itch.
“It’s like she painted—gang signs,” he said softly. “All over the ship.”
That's responsible for the zone of formless terror and also why closing his eyes helped.
