[Continuing coverage of the UN’s 2015 conference on killer robots. See all posts in this series here.]
Things go wrong, with technology and with people. Case in
point: this year, I arrived in Geneva on time after a three-leg flight, but
last year’s trip was a surreal adventure. United’s hopelessly overworked agents
didn’t inform me that my first destination airport was closed as I waited for
the flight, then lied about the unavailability of alternative flights, all
while attempting to work a dysfunctional computer system — followed by a plane
change due to mechanical problems, and then another missed connection.
So yes, things go wrong, with technology and with people,
and even more so with vast systems of people enmeshed with machines and
operating close to some margin determined by the unstable equilibria of
markets, military budgets, and deterrence. Sometimes, one man loses his mind
and scores lose their lives; other times, one keeps his sanity and the world is
saved from a peril it hardly knew.
On September 26, 1983, the Soviet infrared satellite
surveillance system indicated an American missile launch; the computers had
gone through all their “28
or 29 security levels” and it fell to Russian air-defense lieutenant colonel
to decide that it had to be a false alarm, given the small number of missiles
the system was indicating.
This incident occurred just three weeks after
another military-technical-human screw-up had led to the destruction of Korean Air
Lines flight 007 by a Soviet fighter, during one of the most tense years of
the Cold War, and at a time when the Kremlin seriously feared an American first
Petrov was the subject of a 2014 documentary, The Man Who
Saved the World, but if you google that title you may find another
film about another man, Vasili Arkhipov, who
in 1962, during the Cuban Missile Crisis, was the only one of three officers on
board a Soviet submarine to veto a decision to launch a nuclear torpedo at
American ships. The Americans had detected the sub and were dropping depth
charges as a signal for it to surface.
But of course, there have been many “men who saved the world.”
During the 1962 crisis alone one could also cite the role of Llewellyn Thompson,
who had been President Eisenhower’s ambassador to the Soviets, as well as the
role of Kennedy for trusting Thompson’s calm advice, and of Khrushchev, whose memoirs
described how deeply he’d been shaken by nuclear test films he’d seen on joining
the top leadership. And Golda Meir is reputed
to have put the kibosh on Moshe Dayan’s impulse to brandish nuclear spears
during the disastrous early days of the Yom Kippur War, so it isn’t only men
who have been in that position.
An entire study could surely be done on this topic, but for
now we might just observe that when a single human being understands that the
fate of all humanity may rest on his or her own decision, such individuals tend
to exhibit a level of caution, if not wisdom, that may be lacking in even the
people around them. Petrov wasn’t certain the missile launch indication was
false, and he was supposed to go by the “computer readouts” and report an
attack warning up the chain of command. Kennedy’s senior military leadership
unanimously advised him to launch an immediate attack on the Soviet missiles in
Cuba and invade the island, which we now know would have been defended with
tactical nuclear weapons. Arkhipov had to stare down two other high-ranking
officers as the three of them sat for hours in the sweltering heat of a
crippled sub, with depth charges going off all around them. That’s how and why
But this all shows something else: if we build an automated
military system that works perfectly, and carries out doctrines and protocols to
the letter, we may find that we have removed the pin that up to now has kept
the wheels of war from spinning out of control — the simple fact that, if you
are a human being, it’s never a good day for the entire world to die, no matter
what the rulebook says. You always look for a glimmer of hope to avoid an
This, to my mind, is the most compelling reason why we must
draw a clear red line against the further automation of conflict and
confrontation. Human control of technology, and especially of weapons, must be
asserted as an absolute principle, and we have to be clear about what decisions
we are not going to delegate to machines. The direct control of violent force
must be reserved to human beings. This is not a “human right to be killed by
other humans”; rather, it is a human right to live in a world where human
beings have not been reduced to targets for machines to dispatch, or mere
collateral damage in wars between artificially intelligent robots, in which
military necessity has driven humankind out of the loop.
These were some of my thoughts as I sat having a beer with
fellow Stop Killer Robots
campaigners at a bar on the Rhône (actually a dive on a rather dank canal in
one of Geneva’s seedier districts). One of my colleagues had mentioned Petrov,
who reportedly is now a frail, poor, lonely old man. Saving the world can be a
thankless task. Arkhipov died from radiation, Kennedy was shot, Khrushchev
deposed, and Thompson retired into obscurity. Most of my Stop Killer Robots
colleagues struggle to make ends meet, and I had to beg my way to Geneva. Whether
we’ll be of any use in world-saving remains to be seen.
Monday, April 13, 2015
Charles T. Rubin, New Atlantis contributing editor.
Adam Keiper, New Atlantis editor.
Ari N. Schulman, New Atlantis senior editor.
Brendan Foht, New Atlantis assistant editor.
Mark Gubrud, Futurisms contributor.
by Charles T. Rubin
- Machine Morality and Human Responsibility
- Beyond Mankind
- Why Be Human?
- Our Bodies, Ourselves
- The Rhetoric of Extinction
- Man or Machine?
- Artificial Intelligence and Human Nature
by Adam Keiper
by Adam Keiper and Ari N. Schulman
by Ari N. Schulman
by Mark Gubrud
by other authors
- Humanism and Transhumanism (Fred Baumann)
- The Trouble with the Turing Test (Mark Halpern)
- Disenchanting Determinism (Caitrin Nicol)
- The Anti-Theology of the Body (David B. Hart)
- Ageless Bodies, Happy Souls (Leon R. Kass)
- Transitional Humanity (Gilbert Meilaender)
- Till Malfunction Do Us Part (Caitrin Nicol)
- Methuselah and Us (Diana Schaub)
"There's Plenty of Room at the Bottom" 1984 2001 30 Rock Aaron Saenz Abraham Lincoln academia addiction Adi Robertson Adventures of Huckleberry Finn advertising aesthetics Agents of Shield Agnes Heller AI AirLand Battle AirSea Battle Al Jazeera Alan Jacobs Alan Rubenstein Alasdair MacIntyre Alcor Aldous Huxley Alex Backer Alex Knapp aliens Allen Buchanan Alta Charo Amy Gutmann Ana Maria Cuervo Anders Sandberg Andrew Hessel animal uplift Anna Salamon anti-progress Apple argument from inevitability argument from infallibility Ari N. Schulman Aristotle arms control art Arthur C. Clarke artifacts Artificial intelligence artificial life artificial wombs Asilomar assisted reproductive technology assumption of mediocrity Aubrey de Grey Audrey Hepburn augmented reality authenticity automation autonomous weapon system autonomy Avatar avian flu AWS beauty behavioral science Ben Goertzel Benjamin Storey Beyond Therapy Big Dog Bill Joy bioart bioethics bionics body image body modification Brad Templeton Bradley Allenby Bradley J. Thames brain in a bottle Brain Preservation Foundation brain scans brain uploading brain-computer interfaces Brandon Keim Brave New World breathing Brian Christian Brian Malow Bryan Caplan C.S. Lewis Campaign to Stop Killer Robots Caprica Carl Woese cats CCW cell phones character Charles Darwin Charles T. Rubin Charles Taylor chess children chimeras China Christianity Christine Rosen Christmas Claremont Review of Books Clearpath Robotics cloning CNN coercion cognitive computing cognitive enhancement cognitive liberty comments commercials communication technologies compression computational biology Condorcet consciousness constant connection Constiution creativity CRISRP cryonics Cuban Missile Crisis cyborg Cynthia Kenyon Dale Carrico Daniel Sarewitz Daniel Sportiello Darlene Cavalier DARPA Darwinism David A. Noebel David Benatar David Brin David Chalmers David F. Noble David Foster Wallace David Gelernter David Pearce David Rose death Deep Blue democracy Derek Parfit design designer babies despair despotism dictators disability disruption distraction distributive justice diversity DIYbio DOD Down syndrome drones dualism e-memory e-readers Earth eclipse Eclipse of Man economics Ed Boyden Ed Regis efficiency Eliezer Yudkowsky ELIZA embodiment empathy enhancement Enlightenment entropy environmentalism equality of access Eric Drexler Eric Talbot Jensen ethics ETI eugenics everyday life evolution evolutionary psychology Ex Machina existential risks exoplanets extraterrestrial intelligence extraterrestrial life extropy eyes in the back of your head Facebook faith fantasy fashion faux caution fiction Fight Aging First Amendment Fixed Flannery O'Connor flash crash flash war folk psychology Foresight Institute Fort Hood Frances Willard Francis Bacon Francis Fukuyama Frankstein Fred Baumann free markets French Revolution Friedrich Nietzsche friendly AI Future of Life Institute futurism Futurism (art) Futurisms futuristic distance G.K. Chesterton gaming Garry Kasparov Gary Drescher Gary Marcus Gary Wolf gene editing genetic engineering genetics geoengineering George Dvorsky George Orwell germline gene therapy Gilbert Meilaender Gizmodo global warming gloomy God Gödel Golda Meir goodness Google Gordon Bell government GPS Great Mambo Chicken and the Transhuman Condition Greg Benford Gregor Wolbring Gregory Benford H.G. Wells H+ magazine H+ Summit 2010 Halloween Hank Hyena Hans Jonas Hans Moravec heartbreak Heather Knight Heidegger Her history holism hope hubris human cloning human excellence human extinction human life human nature Human Rights Watch human significance humanism humanities humanoid robotics humility humor Iain M. Banks Ian Pearson ICRAC IEET immortality infanticide IQ Irfan Khawaja Isaac Asimov Isaac Newton Issues in Science and technology Itamar Arel IVF J. Craig Venter J. M. Bernstein J. Robert Oppenheimer J.B.S. Haldane Jamais Cascio James Hughes James Jorasch James W. Wagner Jamie Metzl Japan Jason Furman Jason Robert Jeffrey Thurnher Jende Andrew Huang Jeopardy Jesse Schell Jessica Scorpio JFK Jill Lepore John Harris John Markoff John Ruskin John Singer Sargent John Smart Jonah Lehrer Joseph Weizenbaum journalism joy Judaism Juergen Schmidhuber Julian Savulescu jus ad bellum jus in bello just war theory KAL007 Kathleen Lawand Katja Grace Ken Hayworth Kenneth W. Anderson Kevin Jain Kevin Kelly killer robots Kindle kissing Kyle Munkittrick L5 Society lambda calculus Lauren Silbert law law of accelerating returns LAWS LAWS2015 lay science Leon Kass Leon R. Kass Lepht Anonym liberalism libertarian transhumanism libertarianism life extension lifelogging linguistics Lisa Katayama literature Lockheed Martin loneliness Lost love LRASM Ludwig Wittgenstein MacIntyre Conference mainstream Malthusianism man as beast Marc D. Guerra Marcus Hutter Marilynne Robinson Mark Blitz Mark Gubrud Mark Walker Martine Rothblatt Marvel Marvin Minsky Marxism Maryanne Wolf materialism Matthew Crawford Matthew Waxman Max More memory Methuselah Foundation Methuselarity METI Micah Mattix Michael Anissimov Michael Nielsen Michael Pollan Michael Schmitt Michio Kaku Mike Treder Milan Kundera military Millie Ray mind as computer Mind Children mind control minds Modern Times modesty molecular manufacturing monstrosities moral relativism morality morphological freedom Morris Johnson Moshe Dayan movies multitasking nanotechnology NASA Natasha Vita-More Nathaniel Comfort National Geographic National Nanotechnology Initiative natural law natural rights Neal Stephenson Ned Seeman neuro-everything neurobiology neuroelectronics neuroengineering neuroscience Never Say Die conference New America Foundation Nicholas Carr Nick Bostrom Nick Carr Nietzsche Nikita Khrushchev Nikki Olson Nikolai Fyodorov Noah Goodman Noel Sharkey normativity nuclear weapons Oculus Rift Olympics ontological fortitude Orphan Black Orwells Outline of History Oxford Uehiro Centre for Practical Ethics P. W. Singer paradox of choice parenthood Patrick Hopkins Patrick Lin Patrick McGuire patriotism pattern-identity Paul Ramsey Paul Scharre personal identity personhood Peter A. Lawler Peter Singer Peter Thiel philosophy of mind photography Pioneer plastic surgery plastination Plato politics positive sum game postmodernism predation President's Council on Bioethics Presidential Commission for the Study of Bioethical Issues progress progressivism psychology public relations quantum computing racism Radiohead Ramez Naam Randal Koene rationality Ray Kurzweil Ray Tallis recommended reading regulation religion relinquishment René Descartes repugnance resentment watch response to critics resurrection rhetoric of inevitability Richard Feynman Rick Weiss rights ritual Road to Wigan Pier Robert Ettinger Robert P. George Robin Hanson robotics robots Rodney Brooks Roger Scruton romance Ron Bailey Ron Fouchier Russia Ryan Gariepy S. Jay Olshansky Sagan Santa Claus science science fiction scientific enterprise scientism scientists secrecy sectarianism seduction community self-driving cars semi-autonomous September 11 Seth Lloyd SETI sex sex selection sexual enhancement Sherry Turkle simulation Singularitarianism Singularity Singularity Hub Singularity Summit Singularity University Slate sleep smart phones SMBC social interaction social robotics society Socrates Sonia Arrison Sorites paradox space space colonization space exploration sports Stanislav Petrov Star Trek Star Wars stem cell research Stephen Cave Stephen Hawking Stephen Johnston Stephen Wolfram Steve Jobs Steve Sailer Steve Talbott Stuart Hameroff Stuart Russell substrate chauvinism suffering superstition suspended animation synthetic biology systems tacos Tao Tea Party movement Techno-Human Condition Ted Fishman Ted Goertzel Teddy Ruxpin Terminator terrorism The New York Times The New Yorker The Prospect of Immortality the rhetoric of extinction thinking Thomas Malthus thought experiments Tim Tyler Time magazine Tocqueville Todd May Total Recall totalitarianism Transcendence transhumanism transhumanist tech fail translation travel truth Turing Machines Turing Test TV Twitter Tyler Cowen tyranny U.S. government U.S. military UAVs UGVs uncanny valley unemployment United Nations United States government uploading USSR USVs utilitarianism Utopia UUVs Vasili Arkhipov Vatican verification Vijay Kumar virtual reality virtue virtues Voyager Wafaa Bilal Walker Percy Walter Kirn Washington Post Watson we are already x whole-brain emulation William Boothby William Dickens Winwood Reade Wired wisdom Witherspoon Council women's lib work XKCD Yom Kippur War Yuval Levin Zeitgeist Zoltan Istvan
- Killer robots, international law, and just war the...
- Killer Robots, the Free Market, and the Need for L...
- Killer Robots: How could a ban be verified?
- Killer Robots: The Arms Race and the Human Race
- Killer Robots, Human Responsibility, and a Reason ...
- Killer Robots: Where Is the World Heading?
- Blogging the UN Killer Robots Meeting
- ▼ April (7)
- ► 2014 (9)
- ► 2012 (38)
- ► 2011 (46)
- ► 2010 (83)