70

In the worlds of Asimov's robots, what's to stop a bunch of vandal humans from going around telling robots to destroy themselves? According to the Second Law, they would have to obey. Here's a refresher of the laws, copied from Wikipedia:

  1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  2. A robot must obey the orders given to it by human beings, except where such orders would conflict with the First Law.
  3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Even if you suppose a robot is ordered by a human not to destroy itself (say, by a forward-thinking engineer in the factory)…

Robot, assume it will harm a human or humans if you arbitrarily destroy yourself for no reason other than being told to do so.

…wouldn't that pretty much constitute an implied law exists between 2 and 3?

15 Answers 15

27

The Wikipedia article does say that

Asimov himself made slight modifications to the first three in various books and short stories to further develop how robots would interact with humans and each other. In later fiction where robots had taken responsibility for government of whole planets and human civilizations

Having said that, your Denial-of-Service attack idea is interesting and would be useful when thinking about autonomous systems being designed today or in the fictional context, when writing plot structures relevant to current ideas.

This subject has been studied in robotics -- you'll find this space.com article an interesting read:

Science Fiction’s Robotics Laws Need Reality Check

by Jeremy Hsu, Special to SPACE.com - August 19, 2009.

Referring to David Woods, a systems engineer at Ohio State University.

Woods worked with Robin Murphy, a rescue robotics expert at Texas A&M University, to create three laws that recognize humans as the intelligent, responsible adults in the robot-human relationship. Their first law says that humans may not deploy robots without a work system that meets the highest legal and professional standards of safety and ethics. A second revised law requires robots to respond to humans as appropriate for their roles, and assumes that robots are designed to respond to certain orders from a limited number of humans.

The third revised law proposes that robots have enough autonomy to protect their own existence, as long as such protection does not conflict with the first two laws and allows for smooth transfer of control between human and robot. That means a Mars rover should automatically know not to drive off a cliff, unless human operators specifically tell it to do so.

  • 3
    Interesting. It is very telling that the revised laws remove the explicit prohibition against harming humans. By claiming humans must deploy robots meeting "the highest legal and professional standards of safety and ethics", the revised laws leave the door open for military robots (I'm sure military organizations all over the world consider at least some of its weapons "ethical"). – Andres F. Feb 8 '14 at 2:02
  • @AndresF. We found the first law as Asimov wrote it to be utterly unimplementable without first solving strong AI. We do not possess strong AI. – Joshua Sep 19 '16 at 17:58
  • 1
    @Joshua What...? Who's "we"? Asimov's stories already presuppose AI of some kind. And how is your comment related to this answer or my own comment? – Andres F. Sep 19 '16 at 22:45
  • @AndresF: We who had to consider how to build them for real. – Joshua Sep 19 '16 at 22:47
  • 1
    @Joshua So "we" -- meaning "you" -- build robots and find the Three Laws impractical? Um, sure. Are you aware you're in a scifi website? This is like you telling me that you tried psycohistory (or mental powers, or faster than light travel) and found it "impractical". Um. Why is this you're telling me relevant? – Andres F. Sep 21 '16 at 3:47
98

In the story Runaround, Asimov clearly indicates that the Three laws are not purely a boolean logic system. In that story SPD is confused because he hits a balance between the two 'potentials': one from the casually stated order to go and get essential minerals (second law), one by the 'third law' desire to protect his existence, since the area the minerals are in is dangerous to him. The 'second law' potential is weak (because the order was given casually) and the 'third law' potential has been heightened because Speedy is an extremely expensive robot. This leads us to believe that the laws are not hard-and-fast boolean conditions, but a weighted balance of priorities.

The likely answer to your question is therefore that it depends on the circumstances. A casual instruction from someone the robot didn't know might be ignored. A forcefully given instruction from someone in authority might be obeyed.

  • 1
    Also, earlier given commands carry stronger potential than later given commands – Vincent Vancalbergh Sep 18 '13 at 11:32
  • 3
    Sounds like a "fuzzy logic" system where you assign values from 0% to 100% (entirely off or false to entirely true). – Warren P Sep 18 '13 at 19:21
17

Some of the stories set inside the universe handle this case:

  1. In Runaround there is a conflict between the second and third law (check the answer by DJClajworth)

  2. In the Bicentennial Man / The Positronic Man one of the elements of the storyline is that people order the protagonist to dismember himself, which he has to do.

  3. In the second robot trilogy (Caliban, Inferno and Aurora), which was not written by Asmiov one of the main point in the storyline is, that any capable man can ask a robot to destroy himself, and that's what makes a scientist to create a new set of laws, that make the robots more of an ally instead of a slave.

  • Wait Bicentennial Man is by Asimov? No wonder I liked that movie so much. – Michael Brown Sep 18 '13 at 1:31
  • 4
    @MikeBrown I suspect he is talking about the short story. The movie is loosly based on Asimov. – Zoredache Sep 18 '13 at 2:09
  • 2
    @MikeBrown There is a short story called The Bicentennial Man, which was later adopted to a whole novel called The Positronic Man by Robert Silverberg (co-authored with Asimov). The film (called The Bicentennial Man) was mostly based on the latter work. – SztupY Sep 18 '13 at 7:47
  • 2
    @SztupY: If by "mostly" you mean "loosely," then yes. It wasn't as much of a "somewhere Isaac Asimov is rolling over in his grave" butchery as the "I, Robot" movie, but it was still pretty far from the original story... – Mason Wheeler Sep 18 '13 at 16:33
12

A bit late, but I just read this very relevant part from The Robots of Dawn (chapter Daneel). Elijah Baley suggests ordering self-destruction as a way a "roboticide" might have been committed to Daneel. His reaction:

On Aurora - or on any of the Spacer worlds - robots are regarded more highly than on Earth, and are, in general, more complex, versatile, and valuable. The Third Law is distinctly stronger in comparison to the Second Law on Spacer worlds than it is on Earth. An order for self-destruction would be questioned and there would have to be a truly legitimate reason for it to be carried through - a clear and present danger.

11

You are harmed if a vandal takes a sledge hammer and destroys your robot, aren't you? You suffer financial loss; perhaps you even grieve for a lost friend.

If a robot is instructed to destroy itself without motive it would cause you the same harm, and hence feel compelled not to obey.

  • Yes. Not to hyperextend the question, but is a robot always owned? If a robot is not owned, will he destroy himself if ordered to do so without good reason? – kojiro Sep 17 '13 at 16:45
  • 3
    Not since the Bicentennial Man. – DJClayworth Sep 17 '13 at 17:12
  • 2
    If memory serves, all robots were leased (or loaned); the corporation retained ownership so that they could replace any units not behaving "within parameters". – Bevan Sep 18 '13 at 9:59
  • I don't believe the three laws consider financial harm to be 'harm'. In every case I'm aware of it means physical harm. The only exception is one case where the a robot decides that 'emotional harm' is also harm, but from the story it's clear that this is an extension that wasn't previously considered. – DJClayworth Mar 10 '17 at 14:06
6

Yes, you can tell your robot, or rather any robot, to destroy itself. The Bicentennial Man was already briefly mentioned by SztupY, but I'd like to expand on his answer, because this story addresses exactly the scenario described by the OP: what's to stop a bunch of vandal humans from going around telling robots to destroy themselves? Apparently nothing:

(emphasis by me)

The tall one said, “Just lie there.” He said to the other, “We can take him apart. Ever take a robot apart?”

“Will he let us?”

“How can he stop us?”

There was no way Andrew could stop them, if they ordered him not to resist in a forceful enough manner. The Second Law of obedience took precedence over the Third Law of self-preservation. In any case, he could not defend himself without possibly hurting them and that would mean breaking the First Law. At that thought, every motile unit contracted slightly and he quivered as he lay there.

The tall one walked over and pushed him with his foot. “He’s heavy. I think we’ll need tools to do the job.”

The nose said, “We could order him to take himself apart. It would be fun to watch him try.”

Andrew Martin is only saved because a member of the Martin family comes along. As a result of this incidence, the Martins fight for robot rights, and eventually they are successful:

“[...] if the robots have Three Laws to protect men, is it too much to ask that men have a law or two to protect robots?”

[...] in the end a law passed which set up conditions under which robot-harming orders were forbidden.

So after these events, ordering a robot to destroy itself would be illegal. Whether or not this would stop vandals is another question.

3

By killing itself a robot may break the first law. If your robot's job (or at least part of it) involves protecting you, or otherwise looking after you in any way, an order to kill itself would render it unable to do so; that's a clear first law violation so the robot could choose to disobey the order.

2

The purpose of the laws is to protect humans from robo-harm, not the other way around.

Of course many other rules can exist to help the bots respond sensibly to human input, but that's not Asimov's problem.

2

I think it is pretty clear that a person can order a robot to destroy itself, and the robot would comply.

IIRC there is an example of this in the novel The Caves of Steel. There is a robot that had knowledge the murderer didn't want disclosed, and IIRC the murderer simply told the robot to destroy itself. The robot took some sort of radiation-emitting device and held it up to its head, scrambling the positronic brain.

We didn't see the actual command given. Probably "Robot, hold this thing up to your head and destroy yourself" would have been adequate. Certainly "Robot, a human being will come to harm unless you scramble your positronic brain" would have worked.

IMHO, Asimov's Three Laws of Robotics would not work in the real world. They are chiefly important because they marked the first time a writer even contemplated a framework of safety rules to keep robots from becoming a threat... before Asimov's robot stories, robots tended to be dangerous things in SF stories.

Asimov himself wrote a bunch of stories exploring the boundaries of the Three Laws. For example, in The Naked Sun, someone pointed out that you could put a robot brain in a spaceship with no crew on board (a purely automated spaceship) and tell the robot brain that all spaceships are machines with positronic brains, and any radio broadcasts from other ships are untrustworthy... then you would have a robotic space ship that would be able to destroy other space ships despite the First Law.

Other authors poked at the Three Laws in stories. The one I remember best is "A Code for Sam", where a Three Laws robot is miserable because the Three Laws are impractical in real life. (Example, a human is smoking a cigarette, and the robot feels it must keep the human from harming himself, so the robot must take the cigarette away from the human; but both human and robot are distressed.)

http://en.wikipedia.org/wiki/The_Three_Laws_of_Robotics_in_popular_culture

There is a really excellent web comic called Freefall that is actually a hard SF story and which has been exploring the issues surrounding robots and the laws under which robots operate. One of the main characters, perhaps the main character, is Florence Ambrose; she is an uplifted wolf, and legally her status is exactly like a robot (they refer to her as a "biological AI" or just an "AI" sometimes). In the Freefall comic, there are about 20 thousand humans living with about 450 million robots on a world that is being terraformed.

At one point, Florence discusses hierarchy of command: if the Mayor of the town ordered the robots not to destroy themselves without a good reason, then an ordinary person would be unable to countermand the order. Florence suggests making a temporary position that outranks all other positions, then having the person in that position give the robots new orders that in effect give them basic rights. The position could always be re-created at need, if any of the new orders turned out to have problems. It would cut down on abusive treatment toward the robots.

http://freefall.purrsia.com/

The human woman in these three comics is the Mayor. I think the planet just has the one city, so she is the highest-ranked government official on the planet. Also in the room are Florence (the uplifted wolf), and the Mayor's intern.

http://freefall.purrsia.com/ff2200/fc02137.htm

http://freefall.purrsia.com/ff2200/fc02138.htm

http://freefall.purrsia.com/ff2200/fc02139.htm

  • Allowing Robots to judge the quality of the order giver is dangerous - in the story ". . . That Thou Art Mindful of Him" they make robots with a weakened First Law to allow them to determine fitness of the order giver..and they decide that robots are fitter than humans and work to make a world that robots dominate. – Oldcat Jul 2 '14 at 19:40
  • 1
    Yeah, letting the robots use their judgement is dangerous. Not letting them use their judgement is also dangerous... in another Freefall comic, a robotics engineer mutters darkly "We've created a force multiplier for human stupidity!" – steveha Jul 2 '14 at 23:24
2

If you can tell something to the robot? Yes, of course, you can. Whatever.

And what would prevent a vandalism in society full of robots, which are able to understand human orders? Oh yes, and what will prevent a vandalism in current society without robots? It is the same.

More than question to the 3 laws, it is a question about human morale and ethics / and vandalism. 3 laws count with ideal human society, as Asimov hoped, as many other thinkers, that evolution will bring humans to the ideal state/society... but experience told us, evolution will not do this.

Solution is when you think about robots as personal property of someone. If you will destroy or harm personal property of another human, you will receive penalty or go to jail. Way of realization in case of destroyed robots can be different. In case of robots with high value you as the owner can give a priority order. If you destroy your own property, you probably loose guarantee or at least value it brings to you. Both things will prevent humans from doing so... Every concept without personal ownership will go to hell, as everything in current society does the same. Public property without owner are usually in a poor state.

No change to the 3 laws is necessary, as any change could lead to machine's decision about humans as a potential danger and their elimination, as it is described in Terminator saga.

  • L- @user14111: you are right. But destruction or any other misuse should be prevented by something else than this basic laws. And that was why I wrote about ownership and a robot as personal property of someone. // but you are right, 3 laws are too idealistic. If you own something, you can reprogram it as you like. Then you can even order robot to kill a human, and robot will do it. 3 laws was created with ideal vision of ideal society without respect to ownership. And I repeat: it is not question about robots, but about humans, morality, ethics, etc... – Dee Sep 18 '13 at 10:16
  • 1
    The single most unrealistic part about Asimov's Three Laws is the idea that nobody could make a robot that didn't include them. It was stated in the stories that it took a major effort by many people to design a working positronic brain, and it was almost impossible for anyone to duplicate that effort to make a brain that didn't include the Three Laws. Well, in the real world, complex secrets are reverse-engineered all the time... eventually someone would design a simple positronic brain that didn't include the Three Laws. But somehow in Asimov's stories, nobody ever did. – steveha Feb 8 '14 at 1:39
  • L - @ steveha: Yes, you are right. Actually current state is quite opposite, robot development is concentrated in organizations as DARPA, which are clearly connected with military and should help to "save" lives by killing enemies. It is a question of time, when system would be autonomous enough it could be comparable with Skynet. ;c) – Dee Feb 8 '14 at 11:33
  • @steveha: The problem involved wasn't that was impossible; but that such a brain would be inherently unstable. Maybe that's overcomeable with a powerful second law but maybe not. – Joshua Sep 19 '16 at 18:03
  • L- @ Joshua - actually, there will be always technical problems to resolve the 3 laws in real situations and eventually calculate potential loss of the "resolution", which could be in situation (as car acccidents) where any possible solution, including no action will lead to kill humans or destroying itself. – Dee Sep 20 '16 at 8:18
1

well if its true that

it will harm a human or humans if you arbitrarily destroy yourself for no reason other than being told to do so

then by destroying itself it would be breaking rule #1, (regardless of whether it was told so or not) which overrides rule #2. however it depends how you interpret rule #3, does it mean if it does conflict with rule #2 or #1 he must not protect his existence - ignoring all else? or simply that it cannot override rule #1 or #2.

0

Yes, a human can operate most of machines in a way that leads to the destruction of the machine.

To prevent vandalism you describe, robots need authentication system that would allow them to accept orders from authorised persons only. Same way as you cannot just pick and drive a random car in the street or get access to the controls of the train, in reality you should not be able to control a random robot.

Trains and many factory machines have a "Stop" control that can be operated by a random person. Maybe a robot could possibly accept a "Stop" command from any person in case it is safe for other humans to do so.

0

On a basic level yes if you tell a robot to pull it's own head off it will as cited in the example from The Caves of Steel mentioned in another answer though in that case the robot involved was a relatively simple one.

Other robots with a more complex view of the three laws balance them somewhat differently, while the order of the laws remains fundamentally the opposing 'potentials' between the second and third laws would lead a robot to at the very least question the necessity of it's destruction and ignore the order entirely if the need to survive has been impressed upon it strongly enough by another human.

-1

Not if I told my robot to not destroy itsself. It would have to follow that order just as much as the order from the person who told it to.

Now it has two conflicting orders, so it either drops to the 3rd law telling it not to harm itsself, or it has to prioritize the conflicting commands. I think that prioritizing those of it's owner when there is no other way to decide would be a decent default.

-1

Tonight with a friend we discuss about the three laws even the fourth when the zeroth law was added.

We detected the problem. A robot must obey on order given by human, so if a human tell him to kill itself, the robot will do it because the third law only came next.

For the robot not killed itself a simple law must be added to the precedent laws, between to first and second law.

A robot must not end its own existence, except where such orders would conflict with the First Law or cause greater harm to humanity itself

It change all the following laws so it will became :

  • A robot may not harm humanity, or by inaction, allow humanity to come to harm.
  • A robot may not injure a human being or, through inaction, allow a human being to come to harm, except when required to do so in order to prevent greater harm to humanity itself.
  • A robot must not end its own existence, except where such orders would conflict with the First Law or cause greater harm to humanity itself.
  • A robot must obey any orders given to it by human beings, except where such orders would conflict with the First or Second Law or cause greater harm to humanity itself.
  • A robot must protect its own existence as long as such protection does not conflict with the First, Second or Third Law or cause greater harm to humanity itself.
  • The question asks what would happen with the current laws, not how to change the laws to prevent the problem. – Blackwood Sep 11 '17 at 19:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.