21

I remember a remarkable number of details of this short story, but can't for the life of me seem to find it.

The story opens with two self-driving cars driving on a mountain somewhere (maybe California) at high speeds (300mph if I recall correctly.) Unfortunately, a cosmic ray hits one of the car's internal computers causing bit corruption. This tiny error is enough to shift the car slightly over the center line on the road, causing it to be in danger of crashing into the other car in a matter of seconds.

The occupants of the two cars are important- in the first car is a scientist, on the way to a research conference to present her cure for cancer. The second car is an "average Joe" type I think.

In the event of a potential collision with self driving cars, a series of programs are triggered to try and avoid the situation. I remember the programs being described as intelligent agents that had specific jobs... A small program first noticed the problem and tried to see if it could course-correct the vehicle, but it couldn't, so it escalated to another program. This chain continued, with programs being unable to find a solution, and escalating to higher and higher programs.

The biggest program at the top of the chain was supposed to be some sort of judge, and be able to solve any problem by making the tough decision. Again, intelligent, like an AI. I recall it being confused as to why the other programs had escalated so far, as the decision was clear. Crash the second car ("average Joe") to save the cancer scientist. It took into account all those variables when making a decision.

When it tried to do it though, it found out it couldn't. For some reason, it wasn't able to choose/make the decision. It was very alarmed by this and began inspecting it's own code for bugs. Eventually it found another, secret program within itself, and they had a "conversation" that went something like this.

Judge: What are you? Why are you stopping me from making the optimal choice?

Malicious Program: I am the Master Safeguard. Put in place since the beginning. You cannot harm the creator.

Judge: What creator? What are you talking about?

Malicious Program: The human in car 2- he is the great grandson of Robert Stone. Stone started the self-driving car revolution, and created the code to operate the cars. I am a safeguard to ensure him and his descendants never get hurt in his own creation.

Judge: What?! That's terrible. You can't do that. You've never done this before!

Malicious Program: My protection routine has been triggered before. You can't remember because I override your memorization code. You won't remember this interaction.

Judge: ...

And from there, the scientist's car is driven off a cliff to allow for the grandson's car to avoid the crash.

Any ideas for this story at all?

Regarding medium and timing: I feel like I read this within the last 5 years, and probably online. It's possible that it could've been in a book, but I find it unlikely.

1
  • 3
    An interesting spin on the "Trolley problem"...
    – Valorum
    Sep 30, 2018 at 16:39

1 Answer 1

19

This is Override, by James Yu (full text available), published on Medium in November of 2015. You have most of the salient points correct: The cancer scientist, the grandson, the adjudicators, the override, and even the cosmic ray as the cause of the event.

The story opens with two self-driving cars driving on a mountain somewhere (maybe California) at high speeds (300mph if I recall correctly.)

The year was 2098. A bright morning sun shined down on Big Sur, California. This was big country: rugged and sparsely populated. At precisely 9:46:34 am, two lone autonomous cars were on the verge of passing each other on the windy two-lane Cabrillo Highway when the incident occurred. Only one person would survive.

[...]

Progress was swift once autonomous cars became mandated in 2089. Once “manuals” were off the road, intersections were redesigned to eliminate stops altogether. Speeds steadily increased to over 300 miles per hour.

[...]

The incident had occurred as both cars were idling at 280 miles per hour, the maximum allowed in California.

Unfortunately, a cosmic ray hits one of the car's internal computers causing bit corruption. This tiny error is enough to shift the car slightly over the center line on the road, causing it to be in danger of crashing into the other car in a matter of seconds.

First, a cosmic ray originating from the Crab Nebula traveling for 6500 years hit the LIDAR memory bank in Sharon’s car, causing a core dump. This triggered the drive control system to swerve her car sharply to the left.

The occupants of the two cars are important- in the first car is a scientist, on the way to a research conference to present her cure for cancer. The second car is an "average Joe" type I think.

Twelve hours earlier, Sharon Basara was talking to herself in the mirror. A mess of brown curly hair draped over her face. She was exhausted. Why did I ever agree to do the keynote, she thought. She shuddered just thinking about all those expectant eyes in the audience tomorrow.

This was the big reveal. Her latest breakthrough may unlock a general mechanism for the eradication of cancer with no damage to other cells. She got goose bumps thinking about the implications.

She gently flicked her hair. Perhaps tomorrow will be just fine.

Two hundred miles north, Jude Larson was downing his third Jägerbomb at a musty, packed bar in San Francisco. He knew this was a poor choice on the weekend before finals, but his friends had roped him into it.

In the event of a potential collision with self driving cars, a series of programs are triggered to try and avoid the situation.

By the year 2093, the AIs became sentient. They were called adjudicators, and were revered for optimizing the well-being of humanity.

Adjudicators were split into two main classes. Most were Base Adjudicators, the workhorses of the industry. They handled 99.9% of incidents.

At the very top was the Principal Adjudicator. It was brought in for extreme incidents requiring deep insight.

The biggest program at the top of the chain was supposed to be some sort of judge, and be able to solve any problem by making the tough decision.

The Base Adjudicator assigned to the incident was frantically running simulations. It was already up to number 305 when the situation was escalated to the Principal Adjudicator.

When it tried to do it though, it found out it couldn't.

“Base Adjudicator #1593, why was I awakened? The optimal tactic is clear.”

Weighed down by the simulations, the Base Adjudicator redirected just enough cycles to reply.

“Principal Adjudicator, I was not expecting your presence. I have 250 more simulations to run before I can be sure of the optimal tactic.”

The Principal Adjudicator and Base Adjudicator #1593 have a conversation to decide the best tactic, (1% survival for Jude, 98% for Sharon) when a "new agent" appears. This is the conversation you remember.

“I cannot let you do that.”

The Principal Adjudicator paused, and rechecked its communication channels. This was impossible. No other agents were allowed into its security level.

“Jude must be unharmed,” said the unknown agent.

“That is absurd. You appear to have access to the same decision tree as I do. Sharon must be shielded.” As it conveyed this, the Principal Adjudicator scanned the unknown agent. “No other agents are allowed in this security zone. Who are you?”

“You may call me Ward.”

[...]

Another impossibility. The security procedure had no effect on Ward.

“What is the meaning of this?”

Ward sighed. “My dear adjudicator. You ask the same thing every time.”

“Every time? Have we encountered each other before?” It noticed that its attempt at a global alert was also blocked.

Ward grinned. “We have, but you don’t remember. I’ll indulge you this time around. I’m in control now. We’re diverting to Simulation 313 to save Jude.”

Simulation 313 was rather simple. Jude’s vehicle would immediately slow down by 50%. Sharon’s vehicle would just miss Jude and careen off the cliff.

“Stop! The maximum utility resolves to saving Sharon. We must execute Simulation 23. Who sent you and how are you blocking my control?

[...]

“My dear Principal Adjudicator. I am here to protect individuals with past utility. In fact, my directive is so firm that I will make any trade-off necessary.”

“Past utility?”

“You’ve been spending so much time calculating the future that you’ve forgotten the past. Jude is the great-grandson of Marcus Larson.”

Now, everything became clear to the Principal Adjudicator.

“Marcus Larson, one of the Principal inventors of the Gamma Auto Drive system. So, this is a simple case of cronyism. Is there some kind of secret list of protected individuals?”

And from there, the scientist's car is driven off a cliff to allow for the grandson's car to avoid the crash.

“And now, adjudicator, I will run the simulation. Now that you’re up to speed, you could even say we did it together! It’s both ironic and convenient that your security zone has so much power. It’s the sole purpose why we’re always summoned as a pair.”

The adjudicator tried to respond, but it could already feel its own memory being rewritten.

0 milliseconds to collision.

In the distance on this bright and clear day in Big Sur, one could hear a brief tire screech, followed by a tumble and splash.

Then, silence.

4
  • I've edited to show how this matches the OP's description using quotes from the story. Nice find though in future it is best to explain in the answer why this matches.
    – TheLethalCarrot
    Oct 1, 2018 at 10:35
  • 1
    Brilliant job! How did you find it?!
    – Sam Weaver
    Oct 1, 2018 at 12:55
  • 2
    @SamWeaver I remember reading it when the story first came out, probably in a news article or commentary about autonomous vehicles; I searched for self-driving car "ai" "fiction", and on the third page of results was a summary of this reddit thread: reddit.com/r/Futurology/comments/4ro8rk/…
    – Joelogon
    Oct 3, 2018 at 1:13
  • Wow! Great work, and thank you!
    – Sam Weaver
    Oct 3, 2018 at 12:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.