Cambridge’s new existential risk project, and other killer robot news

This past week seems to have been a really good week for the popular media noticing the very important coming threat of killer robots. This is in large part due to a bunch of media outlets noticing Cambridge’s Centre for the Study of Existential Risk–the notice on its website is dated April 2012, but a bunch of news outlets seem to be just noticing it now, including the BBC, the Daily Mail, and the Huffington Post.

The longest article I’ve seen on this, though, is one Bryan Appleyard posted on his own website. It would be a pretty good article except that, as Luke points out, it gets Eliezer Yudkowsky’s views completely wrong. Still, there are good quotes in there:

“You can’t usually control someone who is more intelligent than you,” says Jaan Tallinn, “so, as we develop something that is potentially better at developing further technologies, we might spark this runaway process that consumes all our resources without us having further control”.


“When you think of the forms intelligence might take,” Price says, “it seems reasonable to think we occupy some tiny corner of that space and there are many ways in which something might be intelligent in ways that are nothing like our minds at all.”

The writer Douglas Adams caught this idea in one brilliant phrase when he imagined a “super-intelligent shade of the colour blue”. Funny though this sounds, it is realistic. Our minds are the way they are because of the accident of our biological form and because of human society. An alien machine — or, more likely, one of our own — that had decided to program itself would be formed by neither of those things; it might be just, well, blue. Assuming it was much more intelligent than we are, it would see us as David Attenborough sees some weird creature in the rainforest — as a zoological oddity whose view of the world we could not begin to imagine.

The past week also saw the group Human Rights Watch release a report calling for a ban on fully autonomous military robots, while the Pentagon released new instructions that also seem to be against giving robots too much autonomy.

Wired cover this story under the headline “Pentagon: A Human Will Always Decide When a Robot Kills You,” but the details are murkier:

Laudable impulse, wrong solution, writes Matthew Waxman. A former Defense Department official for detainee policy, Waxman and co-author Kenneth Anderson observe that technological advancements in robotic weapons autonomy is far from predictable, and the definition of “autonomy” is murky enough to make it unwise to tell the world that it has to curtail those advancements at an arbitrary point. Better, they write, for the U.S. to start an international conversation about how much autonomy on a killer robot is appropriate, so as to “embed evolving internal state standards into incrementally advancing automation.”

Waxman and Anderson should be pleased with Carter’s memo, since those standards are exactly what Carter wants the Pentagon to bake into its next drone arsenal. Before the Pentagon agrees to develop or buy new autonomous or somewhat autonomous weapons, a team of senior Pentagon officials and military officers will have to certify that the design itself “incorporates the necessary capabilities to allow commanders and operators to exercise appropriate levels of human judgment in the use of force.” The machines and their software need to provide reliability assurances and failsafes to make sure that’s how they work in practice, too. And anyone operating any such deadly robot needs sufficient certification in both the system they’re using and the rule of law. The phrase “appropriate levels of human judgment” is frequently repeated, to make sure everyone gets the idea. (Now for the lawyers to argue about the meaning of “appropriate.”)

So much for SkyNet. But Carter’s directive blesses the forward march of autonomy in most everything military robots do that can’t kill you. It “[d]oes not apply to autonomous or semi-autonomous cyberspace systems for cyberspace operations; unarmed, unmanned platforms; unguided munitions; munitions manually guided by the operator (e.g., laser- or wire-guided munitions); mines; or unexploded explosive ordnance,” Carter writes.

Finally, in an article for the New Yorker, NYU psychology professor Gary Marcus uses the Human Rights Watch report as a jumping-off point for a fairly good discussion of these issues. He begins by pointing out that self-driving cars could make these issues very pressing far sooner than many of us realize:

Within two or three decades the difference between automated driving and human driving will be so great you may not be legally allowed to drive your own car, and even if you are allowed, it would be immoral of you to drive, because the risk of you hurting yourself or another person will be far greater than if you allowed a machine to do the work.

That moment will be significant not just because it will signal the end of one more human niche, but because it will signal the beginning of another: the era in which it will no longer be optional for machines to have ethical systems. Your car is speeding along a bridge at fifty miles per hour when errant school bus carrying forty innocent children crosses its path. Should your car swerve, possibly risking the life of its owner (you), in order to save the children, or keep going, putting all forty kids at risk? If the decision must be made in milliseconds, the computer will have to make the call.

Marcus also has a nice discussion of why programming a good ethical system is so hard. Commenting in Asimov’s Three Laws:

The trouble with these seemingly sound laws is threefold. The first is technical: at least for now, we couldn’t program a machine with Asimov’s laws if we tried. As yet, we haven’t figured out how to build a machine that fully comprehends the concept of “dinner”, much less something as abstract as “harm” or “protection.” Likewise, we are a long way from constructing a robot that can fully anticipate the consequences of any of its actions (or inactions). For now, a robot is lucky if it can predict would happen if it dropped a glass of water. A.I. has a long way to go before laws as abstract as Asimov’s could realistically be encoded in software.

Second, even if we could figure out how to do the programming, the rules might be too restrictive. The first and second laws, for example, preclude robots from ever harming other humans, but most people would make exceptions for robots that could eliminate potential human targets that were a clear and present danger to others. Only a true ideologue would want to stop a robotic sniper from taking down a hostage-taker or Columbine killer.

Meanwhile, Asimov’s laws themselves might not be fair—to robots. As the computer scientist Kevin Korb has pointed out, Asimov’s laws effectively treat robots like slaves. Perhaps that is acceptable for now, but it could become morally questionable (and more difficult to enforce) as machines become smarter and possibly more self-aware.

The laws of Asimov are hardly the only approach to machine ethics, but many others are equally fraught. An all-powerful computer that was programmed to maximize human pleasure, for example, might consign us all to an intravenous dopamine drip; an automated car that aimed to minimize harm would never leave the driveway.

In spite of how hard these problems are, the fact that they seem to be starting to get attention in the popular press is encouraging.

"Atomsk - Yes, I think the way I feel about it is normal. I think ..."

Let’s talk about violent pornography
"The Scientific Method works by testing a hypothesis for implications, contradictions, and ridiculous/false results. You ..."

Pulling some devastating punches: a review ..."
"A bit OT: Found this article and it is imo closely related to the issue ..."

Let’s talk about violent pornography
"Just one thing for now, because it takes quite a bit of time to think ..."

Let’s talk about violent pornography

Browse Our Archives

What Are Your Thoughts?leave a comment