Daniel Suarez: The kill decision shouldn't belong to a robot

76,064 views ・ 2013-06-13

TED


Please double-click on the English subtitles below to play the video.

00:00
Translator: Joseph Geni Reviewer: Morton Bast
0
0
7000
00:12
I write fiction sci-fi thrillers,
1
12691
3158
00:15
so if I say "killer robots,"
2
15849
2193
00:18
you'd probably think something like this.
3
18042
2361
00:20
But I'm actually not here to talk about fiction.
4
20403
2555
00:22
I'm here to talk about very real killer robots,
5
22958
2948
00:25
autonomous combat drones.
6
25906
3122
00:29
Now, I'm not referring to Predator and Reaper drones,
7
29028
3627
00:32
which have a human making targeting decisions.
8
32655
3260
00:35
I'm talking about fully autonomous robotic weapons
9
35915
3129
00:39
that make lethal decisions about human beings
10
39044
2654
00:41
all on their own.
11
41698
2417
00:44
There's actually a technical term for this: lethal autonomy.
12
44115
4000
00:48
Now, lethally autonomous killer robots
13
48115
2856
00:50
would take many forms -- flying, driving,
14
50971
3069
00:54
or just lying in wait.
15
54040
2746
00:56
And actually, they're very quickly becoming a reality.
16
56786
3109
00:59
These are two automatic sniper stations
17
59895
2484
01:02
currently deployed in the DMZ between North and South Korea.
18
62379
4137
01:06
Both of these machines are capable of automatically
19
66516
2171
01:08
identifying a human target and firing on it,
20
68687
3524
01:12
the one on the left at a distance of over a kilometer.
21
72211
4324
01:16
Now, in both cases, there's still a human in the loop
22
76535
3589
01:20
to make that lethal firing decision,
23
80124
2372
01:22
but it's not a technological requirement. It's a choice.
24
82496
5413
01:27
And it's that choice that I want to focus on,
25
87909
3093
01:31
because as we migrate lethal decision-making
26
91002
2641
01:33
from humans to software,
27
93643
3109
01:36
we risk not only taking the humanity out of war,
28
96752
3476
01:40
but also changing our social landscape entirely,
29
100228
3526
01:43
far from the battlefield.
30
103754
2224
01:45
That's because the way humans resolve conflict
31
105978
4509
01:50
shapes our social landscape.
32
110487
1733
01:52
And this has always been the case, throughout history.
33
112220
2633
01:54
For example, these were state-of-the-art weapons systems
34
114853
2661
01:57
in 1400 A.D.
35
117514
2079
01:59
Now they were both very expensive to build and maintain,
36
119593
3144
02:02
but with these you could dominate the populace,
37
122737
3240
02:05
and the distribution of political power in feudal society reflected that.
38
125977
3889
02:09
Power was focused at the very top.
39
129866
2687
02:12
And what changed? Technological innovation.
40
132553
3528
02:16
Gunpowder, cannon.
41
136081
1871
02:17
And pretty soon, armor and castles were obsolete,
42
137952
3817
02:21
and it mattered less who you brought to the battlefield
43
141769
2533
02:24
versus how many people you brought to the battlefield.
44
144302
3779
02:28
And as armies grew in size, the nation-state arose
45
148081
3638
02:31
as a political and logistical requirement of defense.
46
151719
3680
02:35
And as leaders had to rely on more of their populace,
47
155399
2376
02:37
they began to share power.
48
157775
1833
02:39
Representative government began to form.
49
159608
2599
02:42
So again, the tools we use to resolve conflict
50
162207
3288
02:45
shape our social landscape.
51
165495
3304
02:48
Autonomous robotic weapons are such a tool,
52
168799
4064
02:52
except that, by requiring very few people to go to war,
53
172863
5168
02:58
they risk re-centralizing power into very few hands,
54
178031
4840
03:02
possibly reversing a five-century trend toward democracy.
55
182871
6515
03:09
Now, I think, knowing this,
56
189386
1757
03:11
we can take decisive steps to preserve our democratic institutions,
57
191143
4352
03:15
to do what humans do best, which is adapt.
58
195495
3979
03:19
But time is a factor.
59
199474
2005
03:21
Seventy nations are developing remotely-piloted
60
201479
2851
03:24
combat drones of their own,
61
204330
2157
03:26
and as you'll see, remotely-piloted combat drones
62
206487
2593
03:29
are the precursors to autonomous robotic weapons.
63
209080
4472
03:33
That's because once you've deployed remotely-piloted drones,
64
213552
2767
03:36
there are three powerful factors pushing decision-making
65
216319
3384
03:39
away from humans and on to the weapon platform itself.
66
219703
4600
03:44
The first of these is the deluge of video that drones produce.
67
224303
5259
03:49
For example, in 2004, the U.S. drone fleet produced
68
229562
3853
03:53
a grand total of 71 hours of video surveillance for analysis.
69
233415
5312
03:58
By 2011, this had gone up to 300,000 hours,
70
238727
4499
04:03
outstripping human ability to review it all,
71
243226
3149
04:06
but even that number is about to go up drastically.
72
246375
3664
04:10
The Pentagon's Gorgon Stare and Argus programs
73
250039
2575
04:12
will put up to 65 independently operated camera eyes
74
252614
3164
04:15
on each drone platform,
75
255778
2038
04:17
and this would vastly outstrip human ability to review it.
76
257816
3303
04:21
And that means visual intelligence software will need
77
261119
2160
04:23
to scan it for items of interest.
78
263279
4048
04:27
And that means very soon
79
267327
1348
04:28
drones will tell humans what to look at,
80
268675
2747
04:31
not the other way around.
81
271422
2497
04:33
But there's a second powerful incentive pushing
82
273919
2473
04:36
decision-making away from humans and onto machines,
83
276392
3383
04:39
and that's electromagnetic jamming,
84
279775
2872
04:42
severing the connection between the drone
85
282647
2236
04:44
and its operator.
86
284883
2814
04:47
Now we saw an example of this in 2011
87
287697
2618
04:50
when an American RQ-170 Sentinel drone
88
290315
2956
04:53
got a bit confused over Iran due to a GPS spoofing attack,
89
293271
4307
04:57
but any remotely-piloted drone is susceptible to this type of attack,
90
297578
5114
05:02
and that means drones
91
302692
2052
05:04
will have to shoulder more decision-making.
92
304744
3620
05:08
They'll know their mission objective,
93
308364
3043
05:11
and they'll react to new circumstances without human guidance.
94
311407
4845
05:16
They'll ignore external radio signals
95
316252
2581
05:18
and send very few of their own.
96
318833
2330
05:21
Which brings us to, really, the third
97
321163
2006
05:23
and most powerful incentive pushing decision-making
98
323169
3862
05:27
away from humans and onto weapons:
99
327031
3342
05:30
plausible deniability.
100
330373
3293
05:33
Now we live in a global economy.
101
333666
2887
05:36
High-tech manufacturing is occurring on most continents.
102
336553
4334
05:40
Cyber espionage is spiriting away advanced designs
103
340887
2914
05:43
to parts unknown,
104
343801
1886
05:45
and in that environment, it is very likely
105
345687
2014
05:47
that a successful drone design will be knocked off in contract factories,
106
347701
4734
05:52
proliferate in the gray market.
107
352435
2170
05:54
And in that situation, sifting through the wreckage
108
354605
2460
05:57
of a suicide drone attack, it will be very difficult to say
109
357065
2960
06:00
who sent that weapon.
110
360025
4400
06:04
This raises the very real possibility
111
364425
2800
06:07
of anonymous war.
112
367225
2935
06:10
This could tilt the geopolitical balance on its head,
113
370160
2614
06:12
make it very difficult for a nation to turn its firepower
114
372774
3491
06:16
against an attacker, and that could shift the balance
115
376265
2848
06:19
in the 21st century away from defense and toward offense.
116
379113
3764
06:22
It could make military action a viable option
117
382877
3124
06:26
not just for small nations,
118
386001
2288
06:28
but criminal organizations, private enterprise,
119
388289
2545
06:30
even powerful individuals.
120
390834
2479
06:33
It could create a landscape of rival warlords
121
393313
3328
06:36
undermining rule of law and civil society.
122
396641
3680
06:40
Now if responsibility and transparency
123
400321
3616
06:43
are two of the cornerstones of representative government,
124
403937
2384
06:46
autonomous robotic weapons could undermine both.
125
406321
4320
06:50
Now you might be thinking that
126
410641
1546
06:52
citizens of high-tech nations
127
412187
2246
06:54
would have the advantage in any robotic war,
128
414433
2703
06:57
that citizens of those nations would be less vulnerable,
129
417136
3633
07:00
particularly against developing nations.
130
420769
4288
07:05
But I think the truth is the exact opposite.
131
425057
3524
07:08
I think citizens of high-tech societies
132
428581
2251
07:10
are more vulnerable to robotic weapons,
133
430832
3729
07:14
and the reason can be summed up in one word: data.
134
434561
4465
07:19
Data powers high-tech societies.
135
439026
3481
07:22
Cell phone geolocation, telecom metadata,
136
442507
3190
07:25
social media, email, text, financial transaction data,
137
445697
3472
07:29
transportation data, it's a wealth of real-time data
138
449169
3532
07:32
on the movements and social interactions of people.
139
452701
3373
07:36
In short, we are more visible to machines
140
456074
3775
07:39
than any people in history,
141
459849
2242
07:42
and this perfectly suits the targeting needs of autonomous weapons.
142
462091
5616
07:47
What you're looking at here
143
467707
1738
07:49
is a link analysis map of a social group.
144
469445
3246
07:52
Lines indicate social connectedness between individuals.
145
472691
3634
07:56
And these types of maps can be automatically generated
146
476325
2880
07:59
based on the data trail modern people leave behind.
147
479205
4715
08:03
Now it's typically used to market goods and services
148
483920
2477
08:06
to targeted demographics, but it's a dual-use technology,
149
486397
4416
08:10
because targeting is used in another context.
150
490813
3360
08:14
Notice that certain individuals are highlighted.
151
494173
2560
08:16
These are the hubs of social networks.
152
496733
3280
08:20
These are organizers, opinion-makers, leaders,
153
500013
3590
08:23
and these people also can be automatically identified
154
503603
2682
08:26
from their communication patterns.
155
506285
2382
08:28
Now, if you're a marketer, you might then target them
156
508667
2146
08:30
with product samples, try to spread your brand
157
510813
2543
08:33
through their social group.
158
513356
2829
08:36
But if you're a repressive government
159
516185
1953
08:38
searching for political enemies, you might instead remove them,
160
518138
4810
08:42
eliminate them, disrupt their social group,
161
522948
2760
08:45
and those who remain behind lose social cohesion
162
525708
3169
08:48
and organization.
163
528877
2621
08:51
Now in a world of cheap, proliferating robotic weapons,
164
531498
3324
08:54
borders would offer very little protection
165
534822
2635
08:57
to critics of distant governments
166
537457
1946
08:59
or trans-national criminal organizations.
167
539403
3646
09:03
Popular movements agitating for change
168
543049
3493
09:06
could be detected early and their leaders eliminated
169
546542
3609
09:10
before their ideas achieve critical mass.
170
550151
2911
09:13
And ideas achieving critical mass
171
553062
2591
09:15
is what political activism in popular government is all about.
172
555653
3936
09:19
Anonymous lethal weapons could make lethal action
173
559589
3997
09:23
an easy choice for all sorts of competing interests.
174
563586
3782
09:27
And this would put a chill on free speech
175
567368
3734
09:31
and popular political action, the very heart of democracy.
176
571102
5308
09:36
And this is why we need an international treaty
177
576410
2914
09:39
on robotic weapons, and in particular a global ban
178
579324
3540
09:42
on the development and deployment of killer robots.
179
582864
3908
09:46
Now we already have international treaties
180
586772
3254
09:50
on nuclear and biological weapons, and, while imperfect,
181
590026
3386
09:53
these have largely worked.
182
593412
2288
09:55
But robotic weapons might be every bit as dangerous,
183
595700
3768
09:59
because they will almost certainly be used,
184
599468
3288
10:02
and they would also be corrosive to our democratic institutions.
185
602756
5027
10:07
Now in November 2012 the U.S. Department of Defense
186
607783
3468
10:11
issued a directive requiring
187
611251
2458
10:13
a human being be present in all lethal decisions.
188
613709
4519
10:18
This temporarily effectively banned autonomous weapons in the U.S. military,
189
618228
4776
10:23
but that directive needs to be made permanent.
190
623004
3753
10:26
And it could set the stage for global action.
191
626757
4376
10:31
Because we need an international legal framework
192
631133
3845
10:34
for robotic weapons.
193
634978
2138
10:37
And we need it now, before there's a devastating attack
194
637116
2928
10:40
or a terrorist incident that causes nations of the world
195
640044
3152
10:43
to rush to adopt these weapons
196
643196
1924
10:45
before thinking through the consequences.
197
645120
3771
10:48
Autonomous robotic weapons concentrate too much power
198
648891
2981
10:51
in too few hands, and they would imperil democracy itself.
199
651872
6283
10:58
Now, don't get me wrong, I think there are tons
200
658155
2686
11:00
of great uses for unarmed civilian drones:
201
660841
2618
11:03
environmental monitoring, search and rescue, logistics.
202
663459
3939
11:07
If we have an international treaty on robotic weapons,
203
667398
2826
11:10
how do we gain the benefits of autonomous drones
204
670224
3587
11:13
and vehicles while still protecting ourselves
205
673811
2648
11:16
against illegal robotic weapons?
206
676459
3980
11:20
I think the secret will be transparency.
207
680439
4741
11:25
No robot should have an expectation of privacy
208
685180
3013
11:28
in a public place.
209
688193
3451
11:31
(Applause)
210
691644
5048
11:36
Each robot and drone should have
211
696692
2045
11:38
a cryptographically signed I.D. burned in at the factory
212
698737
2883
11:41
that can be used to track its movement through public spaces.
213
701620
2923
11:44
We have license plates on cars, tail numbers on aircraft.
214
704543
3381
11:47
This is no different.
215
707924
1841
11:49
And every citizen should be able to download an app
216
709765
2012
11:51
that shows the population of drones and autonomous vehicles
217
711777
3125
11:54
moving through public spaces around them,
218
714902
2429
11:57
both right now and historically.
219
717331
2733
12:00
And civic leaders should deploy sensors and civic drones
220
720064
3548
12:03
to detect rogue drones,
221
723612
2344
12:05
and instead of sending killer drones of their own up to shoot them down,
222
725956
3176
12:09
they should notify humans to their presence.
223
729132
2992
12:12
And in certain very high-security areas,
224
732124
2606
12:14
perhaps civic drones would snare them
225
734730
1909
12:16
and drag them off to a bomb disposal facility.
226
736639
2841
12:19
But notice, this is more an immune system
227
739480
3027
12:22
than a weapons system.
228
742507
1321
12:23
It would allow us to avail ourselves of the use
229
743828
2592
12:26
of autonomous vehicles and drones
230
746420
2032
12:28
while still preserving our open, civil society.
231
748452
4295
12:32
We must ban the deployment and development
232
752747
2999
12:35
of killer robots.
233
755746
1862
12:37
Let's not succumb to the temptation to automate war.
234
757608
4850
12:42
Autocratic governments and criminal organizations
235
762458
2718
12:45
undoubtedly will, but let's not join them.
236
765176
2956
12:48
Autonomous robotic weapons
237
768132
1891
12:50
would concentrate too much power
238
770023
2051
12:52
in too few unseen hands,
239
772074
2482
12:54
and that would be corrosive to representative government.
240
774556
3255
12:57
Let's make sure, for democracies at least,
241
777811
2961
13:00
killer robots remain fiction.
242
780772
2604
13:03
Thank you.
243
783376
1110
13:04
(Applause)
244
784486
4565
13:09
Thank you. (Applause)
245
789051
4616
About this website

This site will introduce you to YouTube videos that are useful for learning English. You will see English lessons taught by top-notch teachers from around the world. Double-click on the English subtitles displayed on each video page to play the video from there. The subtitles scroll in sync with the video playback. If you have any comments or requests, please contact us using this contact form.

https://forms.gle/WvT1wiN1qDtmnspy7