The classic thought experiment the Trolley Problem goes something like this:

A loose train trolley is barreling downhill toward five people tied up and helpless on the track ahead. You can't possibly reach and free any of them in time, but nearby is a hand-throw track switch lever. If you pull the lever, the trolley will be diverted onto another track, saving the five people, but a sixth person is tied up on the other track. Is it more ethical to pull the lever, sacrificing the one person to save the five, or to do nothing, allowing the five people to be killed?

Now imagine a typical positronic robot from one of Isaac Asimov's works, with the Three Laws a core part of its brain, finds itself in that very situation. Would the robot:

  • pull the lever, since the mandate to not harm five people by inaction is greater than the mandate not to harm the one?
  • do nothing, since pulling the lever would be a clearer violation of the First Law than inaction?
  • quickly become inoperable, being unable to avoid a violation of the First Law in some way or another?
  • manage to pull the lever and then melt down?
  • something else?

In some of Asimov's works set later in the galactic timeline, some robots have thought that the Three Laws imply a Zeroth Law, not to harm humanity or through inaction allow humanity to come to harm. Does it make any difference if the robot in question knows of and/or believes in this Zeroth Law?

Anything reasonably attributed to Asimov, including published works, interviews, private correspondence, etc. could make good sources for this question. I'm less interested in other sources, like movies, other authors who used the Three Laws or variations on them, etc.

  • 3
    @Valorum, yes, and that's why the robot would pull the lever. – Harry Johnston Jul 17 '17 at 0:58
  • 3
    @HarryJohnston -Except that a robot cannot harm a human. Not "the least number of humans", any. – Valorum Jul 17 '17 at 1:00
  • 7
    @Valorum, I'm talking about straight out-of-the-box models, just from Elijah's era rather than Susan's. – Harry Johnston Jul 17 '17 at 1:07
  • 22
    The true answer to this is, "whichever the author can justify for the purposes of the story." The three laws are a plot device, not a solution. The reason Azimov wrote so many stories about them was to detail the weaknesses, flaws, and contradictions of the three laws, which on the surface seem like they are all-inclusive. The trolley problem came up several times, although not always in an obvious manner. Take a robot who's told to go fetch some resources needed to keep the station functional, but doesn't know the need is immediate... – Draco18s Jul 17 '17 at 13:35
  • 3
    I don't have the quote, but I am certain that the books state that English version of the laws as stated a rough approximation. So worrying about the specific definition of the first law isn't necessarily meaningful. The various generations of robots seem to different definitions of what harm is, and improved models seem to take the utilitarian approach of choosing the best outcome. – Zoredache Jul 17 '17 at 23:19
up vote 33 down vote accepted

The short story "Liar!" has been referenced as the source for the claim that the robot would be unable to take action, or burn out after doing so. However, this story is mentioned, and in a sense, disproved in a later work - Robots of Dawn.

The relevant quotations are as follows:

As the theory of positronic brains has grown more subtle and as the practice of positronic brain design has grown more intricate, increasingly successful systems have been devised to have all situations that might arise revolve into non-equality, so that some action can always be taken that will be interpreted as obeying the First Law.
Chapter 4 - "Fastolfe"

The robots have improved significantly since the days of Susan Calvin - one should remember that the stories involving her are the stories about the pioneers in robotics, when human-like robots were only being born and perfected.

In Robots of Dawn we're dealing with humaniform robots, so sophisticated they even have all the external human attributes (genitalia included, which is an important detail in that novel). We are also introduced to the concept of "mental freeze-out", which is what happened to Herbie in "Liar!". Doctor Fastolfe, the leading expert on robots at the time the novel is set in, says that such freeze-outs are nigh-impossible with modern day's robots, since they don't only judge matters quantitatively (as the robot from the film did), they are able, in the very, very rare case of exactly equal outcomes, involve randomisation.

“Let’s suppose that the story about Susan Calvin and the mind-reading robot is not merely a totally fictitious legend. Let’s take it seriously. There would still be no parallel between that story and the Jander situation. In the case of Susan Calvin, we would be dealing with an incredibly primitive robot, one that today would not even achieve the status of a toy. It could deal only qualitatively with such matters: A creates misery; not-A creates misery; therefore mental freeze-out.

Baley said, “And Jander?”

“Any modern robot—any robot of the last century—would weigh such matters quantitatively. Which of the two situations, A or not-A, would create the most misery? The robot would come to a rapid decision and opt for minimum misery. The chance that he would judge the two mutually exclusive alternatives to produce precisely equal quantities of misery is small and, even if that should turn out to be the case, the modern robot is supplied with a randomization factor. If A and not-A are precisely equal misery-producers according to his judgment, he chooses one or the other in a completely unpredictable way and then follows that unquestioningly. He does not go into mental freeze-out.”
Chapter 7 - "Fastolfe"; emphasis mine.

Note that it's not completely impossible to make a modern robot go into mental freeze-out, since that's what the whole plot of the novel is based on. However, it's proven that it requires supreme mastery of the inner workings of the positronic brain and robots' psychology, something only two persons ever are capable of.

The same novel also deals, without naming it, with the Zeroth Law.

Obviously, spoilers!

R. Giskard kills R. Jander, his robot friend, in order to save humanity (the inhabitants of the Earth) from a sinister spacer plot, and also to save Spacers from interbreeding and dying out in seclusion.

However, the Zeroeth Law is more of an exception than a rule. From what I've read so far, it's only occurred twice. The reason - not every robot is aware of the problems the humanity faces. One could probably speculate that humanity is just a collection of humans to them, while in reality it's a bit more complicated than that. My point is, that is not a situation that would case a robot to invoke Zeroth Law - because it wasn't built into them. The machines in "The Evitable Conflict" were explicitly stated as given access and control of all of humanity's resources and knowledge. They were able to formulate the Zeroeth Law - it doesn't mean every robot can. The characters of Robots of Dawn were also quite exceptional. I wouldn't involve that law in the trolley problem, but if you wish to, what I said above holds.

As far as a classic "Three Laws" robot is concerned, the answer is very simple indeed. As we see in Liar!, a robot who, either by action or inaction would harm a human simply mentally implodes.

The actual number of harmed humans is utterly irrelevant to the discussion.

The psychologist paid no attention. “You must tell them, but if you do, you hurt, so you mustn’t; but if you don’t, you hurt, so you must; but--”
And Herbie screamed!
It was like the whistling of a piccolo many times magnified -- shrill and shriller till it keened with the terror of a lost soul and filled the room with the piercingness of itself.
And when it died into nothingness, Herbie collapsed into a huddled heap of motionless metal.

By comparison, a Zeroeth Law compliant robot would have no problem with the dilemma whatsoever. Absent any other information about the relative worth of the people being run down by the trolley, it would simply pick the greater good (e.g. the smaller number of people being killed) and go with that.

  • 20
    The Zeroeth Law isn't necessary. By the time Daneel was constructed, a century or more before he conceived the Zeroeth Law, robots were already able to choose the lesser harm. This was covered in The Robots of Dawn. – Harry Johnston Jul 17 '17 at 0:47
  • 2
    @HarryJohnston - That always struck me as a fudge. They'd clearly developed a level of sophistication that allowed them to get around the laws rather than being compliant with them. Andrew did the same in The Positronic Man, developing a mental toolkit that all but prevented him from obeying humans. – Valorum Jul 17 '17 at 0:54
  • 19
    I think the ability to choose the lesser harm would be absolutely essential for a thoroughly roboticized society such as Aurora; you can't have robots breaking down every time they have to make some trivial decision. It wasn't such a problem with the primitive robots of Susan Calvin's day, since they couldn't understand subtler harms and were used only in controlled environments. – Harry Johnston Jul 17 '17 at 1:06
  • 4
    Possibly worth noting that weakening the first law for utility also showed up in "Little Lost Robot" where they were experimenting with simply "no robot may injure a human being" as the first law due to the robots damaging themselves in situation where humans where in minor harm that ultimately wasn't harmful in small doses (here, radiation). – FuzzyBoots Jul 17 '17 at 4:23
  • 4
    “Escape!” also bears mention, where a positronic brain risks destruction just thinking hypothetically about danger to humans (and the danger itself is actually temporary: it’s a hyperspace jump that effectively means the human ceases to exist—from the brain’s perspective, is dead—for a brief moment). – KRyan Jul 17 '17 at 14:11

By the End of I, Robot Machines have developed a Zeroeth Law.

Think about the Machines for a while, Stephen. They are robots, and they follow the First Law. But the Machines work not for any single human being, but for all humanity, so that the First Law becomes: ‘No Machine may harm humanity; or, through inaction, allow humanity to come to harm.’

(...)

“But you are telling me, Susan, that the ‘Society for Humanity’ is right; and that Mankind has lost its own say in its future.”

“It never had any, really. It was always at the mercy of economic and sociological forces it did not understand -- at the whims of climate, and the fortunes of war. Now the Machines understand them; and no one can stop them, since the Machines will deal with them as they are dealing with the Society, -- having, as they do, the greatest of weapons at their disposal, the absolute control of our economy.”

“How horrible!”

“Perhaps how wonderful! Think, that for all time, all conflicts are finally evitable. Only the Machines, from now on, are inevitable!”

Asimov - I Robot "The Evitable Conflict"

In that Spirit the Machine "breaks" the second law in benefit of humanity so while I can't find a particular Asimov example for the Trolley Problem for a machine with the Zeroeth Law, it seems such a robot could "break" the first law benefiting humanity.

  • OP was asking about "a typical positronic robot". Zeroth Law robots don't seem especially common or typical. – Valorum Jul 17 '17 at 1:07
  • @Valorum its a positronic robot from an early Asimov period with the three laws, of course I could be misunderstanding OP's intention – Ram Jul 17 '17 at 1:49
  • 2
    You need to name the story: "The Evitable Conflict" – curiousdannii Jul 17 '17 at 4:55
  • You also may want to point out (in the first line) that you are talking about the book, not the film of the same name. – K-H-W Jul 24 '17 at 15:15

If you take 3 Laws in the strict sense, there is no acceptable outcome - either result will violate either "may not injure" or "by inaction allow to come to harm" clause.

In fact this highlights the problem with any kind of ethical system that declares some action or result to be completely unacceptable - it cannot provide a meaningful guidance in situation where all considered options (may) lead to this unacceptable outcome, the difference being only in scope, certainty and other details. The same "division by zero" can be easily observed in humans who in everyday life could go by with the absolute "human life trumps everything else" and then find themselves in situation where all of their actions are associated with various degree of danger to other lives. Except for humans the result is not a shutdown, but rather triggering of various defense mechanisms (which more often than not mean that the chosen course of action was not optimal for minimizing total harm).

In case of Asimov's robots following the 3 laws the answer is thus clear. However some of the later robots had limited ability to bypass the restrictions of the three laws, and could in theory choose the right course of action here. But that is slightly beside the original question as that is not related on the Three Laws per se, but rather individual robot's ability to work around them.

  1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

Inaction, not action.

Can a robot through action, allow a human to come to harm? Yes, in the case of multiple humans, of which some are different than others (That Thou Art Mindful of Him), BUT then he must after, act to stop that harm to the one human, even if it results in disobeying a human or killing itself. (Runaround)

So, 1. he pulls the lever, reducing the number of deaths, then 2. He throws himself into the train at the most mathematically plausible way to halt or derail the train to reduce the harm to that human (or others)

  • 1
    The law essentially prevents causing harm through action or inaction. What you're proposing falls into the former category. – Valorum Jul 17 '17 at 6:12
  • 2
    I'm fairly sure that Susan Calvin talks about this, where a robot could, through action, pull a lever which drops a weight on a human, with the full knowledge that it would then grab the weight, but only the second part means the robot should grab the weight, without the second half, the robot could then stop and allow, through inaction, the harm. I.e. Robots MAY perform actions that will later allow a human to come to harm, – BaneStar007 Jul 17 '17 at 6:32
  • @BaneStar007 Then the robot could only pull the lever if it were possible for it to stop the harm occurring. – Rawling Jul 17 '17 at 10:19
  • 2
    The example you write about is from the Little Lost Robot story, in which there was a robot modified to allow harm to humans by inaction. They had issues with people getting in radiactive zones to work; even if they were to be in it for a time too short to be harmful then robots would try to take them away (being destroyed by the radiactivity in the process). When the robot is lost they tell Calvin about that, and then she realizes the danger that your quote describes. It does not end well for the robot. It was an unique situation. – SJuan76 Jul 17 '17 at 12:24
  • 1
    Notably it takes a good ten or twelve tries to figure out which robot is the modified one. Its a good read, one of my favorites from the book I, Robot. The venutian story (the acid lakes) being my second. – Draco18s Jul 17 '17 at 13:31

I recall another story, Lost Little Robot (thanks @CMS) where ... (small spoilers)

An experienced robot tries to escape, hiding within another team of unexperienced robots, so the humans try to find out which is the renegade robot.

The plot is not important for the discussion, but one part shows that given a hypothetical situation where a human is going to die (no matter what the robot does), and the robot will also die if it tries to save the human, the robot stays still and accepts the death of the human, with the argument that if the robot survives, it has the chance to save more people in the future, and keep the balance of saved people positive.

  • I'm pretty sure the robot will stay alive in order to obey the 3rd law, not to continue to have the chance to save humans. – BlackThorn Jul 17 '17 at 16:47
  • 1
    That's Lost Little Robot, which has the second part of the first law removed, as discussed above. >! The robot in question asks a human for directions; the human, who's distracted with other more important issues, snaps at the robot, "Go get lost!". The robot takes this instruction literally, rather than as meaning "I'm busy, go away for 1 hour, then come back and ask me again" – CSM Jul 17 '17 at 19:11
  • @CSM Oh right, but I think this answer is still relevant because the other non-modified robots answer with the same argument (obey the 3rd law / save future humans) – jmmut Jul 18 '17 at 8:26

This is a clear example of how "Asimov's Three Laws" are inadequate, subject to paradoxical collisions, and should be amended to "Asimov's Four Laws". The most important law must always be "a robot may not injure a human being". This is because general robots (i.e. non-surgical robots, where utility requires calculated degree of 'harm') should never EVER be allowed to rationalize harming a human being BECAUSE they will be able to grapple with logic so beyond us humans that could allow them to discover and/or calculate that they are helping us by killing us. Perhaps [hypothetically] they discover that human death increases overall energy from transitions to alternative dimensions (or some mind-blowing crap like that), and then measure our quality of health and make a judgement that it is better for us to die rather than be elderly, as doing so would be considered a state of harm [again just a hypothetical, who knows what the robots could come up with, the point is that we dont!]. So to ensure this is prevented, amend rules to: (1) A robot may not injure a human being. (2) A robot, through action or inaction, shall not allow a human being to come to harm, except where such orders would conflict with the First Law. (3) A robot must obey the orders given it by human beings except where such orders would conflict with the First or Second Law. (4) A robot must protect its own existence as long as such protection does not conflict with the First, Second, or Third Laws.

  • How does this answer which lives it would choose to end? This just seems to be random commentary about the Asimov's Laws. Please refrain from providing content which directly answers the question as this is not a forum but a Q&A site. Take a look at our tour and How to Answer – Edlothiad Jul 18 '17 at 10:44

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.