In Ender's Game by Orson Scott Card, Ender ends up playing a special computer game extensively and obsessively.
The stated purpose of the game is to analyze the students' mind.
Yet, the school personnel (Anderson?) pretty openly admits that they don't know why the game shows certain things to Ender.
Why in the world wouldn't an important piece of software like that have detailed debug logs?
The mind game doesn't give debug logs because it's deliberately programmed not to. From Ender's Game, chapter 9, "Demosthenes and Locke":
"OK, it's for his own good. But why. His brother is dangerous, his brother was rejected for this program because he's one of the most ruthless and unreliable human beings we've laid hands on. Why is he so important to Ender? Why, after all this time?"
"Honestly, sir, I don't know. And the mind game program is designed so that it can't tell us. It may not know itself, actually. This is uncharted territory."
Earlier in the same conversation, Colonel Graff and Major Imbu discuss the purpose and meaning of the game. The game is not, strictly speaking, designed for the benefit of those watching. It's designed for the psychological benefit of the children playing it.
"The End of the World in the game isn't necessarily the end of humanity in the bugger wars. It has private meaning to Ender."
[...]
"The mind game is a relationship between the child and the computer. Together they create stories. The stories are true, in the sense that they reflect the reality of the child's life. That's all I know."
The mind game is designed to help the Battle School students deal with their own personal conflicts in a non-destructive way. No one else was able to overcome their internal struggles with the speed and determination that Ender had. The Giant had previously defeated everyone who crossed its path, but Ender overcame it. The Colonel had never seen anyone do that before, but he didn't fully understand what he was seeing after the Giant. The game could tell Ender's next major struggle was with his dark past: Peter. After having that revealed, they knew that Ender would have to revisit his happy past with Valentine in order to overcome the End of his World. Only then would Ender be ready to face the final challenges that would present themselves.
[T]he game is an AI following a sophisticated algorithm, not a simple if-this-then-that approach. Even for a simple neural net pattern recognition algorithm, a detailed debug log simply doesn't tell you anything about why something happened. – Tynam
Debugging logs and debugging protocols are not always a detailed look at why things happened. In a typical small program (1000 lines of code, plus library code - total under 5K lines of code), the debug log only shows data for those breakpoints that were set and for where a detected error occurs...
Even when the program has useful breakpoints, most non-fatal errors occur without throwing an exception (triggering the error reporting code). You know it's an error because the result is unexpected - and then you go back set more breakpoints, recompile in debug mode and then watch the variables in those breakpoints.
And, as Tynan points out, neural nets are complex. A simple neural net simulation runs to 500 lines of code, for a 3-4 node net. A usefully large net would have to be hardware based to be fast enough, running to several tens of thousands of nodes, and debugging that would require large volumes of data, special interfaces, and dedicated teams working on figuring even the smaller portions of the code. Further, the AI would be driving large banks of processors running auxiliary code in non-neural hardware. Further, to get the level of complexity, the system must have an adaptive heuristic model, in addition to a neural net.
Also, unless you reset the hardware completely, an adaptive heuristic neural network is not the same after use as before. The nature of heuristics is that a heuristic program self adjusts based upon a built-in feedback loop. Eurisko was a non-AI adaptive heuristic. The programmer, D. Lenat, claims to have had little idea why Eurisko chose the strategies it did, other than they worked. In iteration, it became the single best Trillion Credit Squadron player in the world... sufficiently good that its entry threatened to cause the game designer to close the tournament.
In the case of Ender and the AI, you lack control over the inputs - the unexpected output is noteworthy but not replicable, as they lacked both the duplicate hardware and the input logs. The logging of input was restricted; much of Ender's was recorded, but that's not enough. To replicate an adaptive heuristic program's status, you need to either snapshot the whole collective input as it goes in, so you can refeed it, or snapshot prior to the point of interesting error and record all input from there on.
Even being able to replicate with modifications, you still don't have the whole picture; Neural networks are subject to random inputs from various radiation altering the conductivity of their connection matrices. In orbit, this is even more potentially profound. Since the Battle School was in orbit, cosmic radiation and solar radiation provide limited random inputs that can't be replicated.
Add to that that mention is made that the Battle School AI was the single most complex program yet deployed. Large programs have other issues, especially since they are done in modules, with various modules developed independently and then merged together by a different team, sometimes in 3-4 layers of combinations. A hidden bug in a module may not throw an exception itself - but may trigger an exception throw in a different module. This is the case that happened with the USS Yorktown in 1997 - a programmer failed to implement a division by 0 check on a calculation field; the bad data was passed before calculation, resulting in a cascade of computer failures. A Similar issue in hardware was had with certain AMD processors in the 1990's- a particular error would shut down a given sub-processor (r-unit) - in testing it was overlooked because the processor was so fault tolerant that it didn't appear to be a hardware failure; once the device hit field use, the failures were more noted.
Now, given the claimed complexity of the Battle School Game program, debugging logs would either be huge (and thus done infrequently) or practically useless, showing only thrown exceptions. Coupled to the Adaptive Heuristics and the Neural Network, debugging becomes a near impossible task. Even replication is impossible, due to lack of totality of logs and the random firing of neural links from cosmic and solar radiation.
And that's before accounting for apparently random series of external inputs - the philotic inputs snuck through the ansibles by the Hive Queens. Those inputs, the initial spark leading to Jane, were not recorded. They only went undetected due to the sheer complexity and massive data input allowances of the Battle School computer systems... but feeding them into the heuristics means that the program can't be replicated separately, as some inputs will be missing.
