They did, but Galen misdirected them

There's a supplemental document included in the Rogue One novelization, comprised of several messages sent between Galen and one of the engineering managers, where we learn that Galen's reactor modifications actually did trigger diagnostic warnings, as one would expect:

I had the droids generate a new Systems Safety and Compatibility Report incorporating your team's proposed adjustments to the reactor core. The new plans triggered a dozen subsystem warnings and spat out one blazing red stain on the line labeled "Hypermatter Annihilator Unit." I didn't bother asking my astromech how bad that could be — a redline on a critical system speaks for itself.

^{Rogue One: A Star Wars Story - Official Novelization Supplemental Data: Battle Station Engineering Notes}

There's some back-and-forth between the assorted parties, but Galen manages to get his reactor changes approved mostly by appealing to Imperial interests in:

• Getting the damn thing actually working, on-schedule (or less behind schedule)

You may be too obtuse to realize it, Erso, but I'm doing you a favor. This project needed to be done weeks ago.

Send me the final venting shaft and exhaust port plans. I'll bypass SSCR and submit them for production, manufacturing, and installation.

Changes have been approved.

^{Rogue One: A Star Wars Story - Official Novelization Supplemental Data: Battle Station Engineering Notes}

• Keeping the officers happy:

I oversaw construction of the northern command sector myself. Tarkin has already toured the facilities. If the particle funnel and recycler can't go anywhere else, stick with options two [a time-consuming research project on improving the reactor technology] and three [an exhaust port].

[...]

Send me your final plans. I'll declare the droids in error and override the next SSCR.

I'm not burying Krennic in redline reports while you figure out how to keep a handful of stormtroopers from developing a cough.

^{Rogue One: A Star Wars Story - Official Novelization Supplemental Data: Battle Station Engineering Notes}

They did, but Galen misdirected them

There's a supplemental document included in the Rogue One novelization, comprised of several messages sent between Galen and one of the engineering managers, where we learn that Galen's reactor modifications actually did trigger diagnostic warnings, as one would expect:

I had the droids generate a new Systems Safety and Compatibility Report incorporating your team's proposed adjustments to the reactor core. The new plans triggered a dozen subsystem warnings and spat out one blazing red stain on the line labeled "Hypermatter Annihilator Unit." I didn't bother asking my astromech how bad that could be — a redline on a critical system speaks for itself.

^{Rogue One: A Star Wars Story - Official Novelization Supplemental Data: Battle Station Engineering Notes}

There's some back-and-forth between the assorted parties, but Galen manages to get his reactor changes approved mostly by appealing to Imperial interests in:

• Getting the damn thing actually working, on-schedule (or less behind schedule)

You may be too obtuse to realize it, Erso, but I'm doing you a favor. This project needed to be done weeks ago.

Send me the final venting shaft and exhaust port plans. I'll bypass SSCR and submit them for production, manufacturing, and installation.

Changes have been approved.

^{Rogue One: A Star Wars Story - Official Novelization Supplemental Data: Battle Station Engineering Notes}

• Keeping the officers happy:

I oversaw construction of the northern command sector myself. Tarkin has already toured the facilities. If the particle funnel and recycler can't go anywhere else, stick with options two [a time-consuming research project on improving the reactor technology] and three [an exhaust port].

[...]

Send me your final plans. I'll declare the droids in error and override the next SSCR.

I'm not burying Krennic in redline reports while you figure out how to keep a handful of stormtroopers from developing a cough¹.

^{Rogue One: A Star Wars Story - Official Novelization Supplemental Data: Battle Station Engineering Notes}

¹ Additional context which I don't want to quote in the interests of brevity: Galen has responded that installing venting shafts will only mostly solve the problem; there will still be some radiation leakage into crewed areas. Upon being assured by Galen that the leakage won't affect command sections of officer's quarters, that message was sent