Daniel Suarez: The kill decision shouldn't belong to a robot

Translator: Joseph Geni Reviewer: Morton Bast

Prevodilac: Miloš Milosavljević Lektor: Tatjana Jevdjic

I write fiction sci-fi thrillers,

Ja pišem naučno fantastične trilere

so if I say "killer robots,"

i ako kažem "roboti ubice"

you'd probably think something like this.

verovatno ćete pomisliti na ovako nešto.

But I'm actually not here to talk about fiction.

Ali nisam ovde da bih pričao o fikciji.

I'm here to talk about very real killer robots,

Ovde sam da bih pričao o vrlo stvarnim robotima ubicama,

autonomous combat drones.

autonomnim bespilotnim letelicama za borbu.

Now, I'm not referring to Predator and Reaper drones,

Ne mislim pritom na bespilotne letelice "Predator i "Riper"

which have a human making targeting decisions.

gde odluke o ciljanju donose ljudi.

I'm talking about fully autonomous robotic weapons

Govorim o potpuno nezavisnom robotskom oružju

that make lethal decisions about human beings

koje donosi odluke o smrti ljudskih bića

all on their own.

potpuno samostalno.

There's actually a technical term for this: lethal autonomy.

Postoji i tehnički izraz za to: smrtonosna autonomija.

Now, lethally autonomous killer robots

Smrtonosno autonomni roboti ubice

would take many forms -- flying, driving,

mogu imati više oblika - leteći, na točkovima

or just lying in wait.

ili samo u zasedi.

And actually, they're very quickly becoming a reality.

U stvari, oni vrlo brzo postaju realnost.

These are two automatic sniper stations

Ovo su dve automatske snajperske stanice

currently deployed in the DMZ between North and South Korea.

trenutno postavljene u demilitarizovanoj zoni između Severne i Južne Koreje.

Both of these machines are capable of automatically

Obe ove mašine su sposobne za automatsko

identifying a human target and firing on it,

identifikovanje ljudskih meta i pucanje na njih,

the one on the left at a distance of over a kilometer.

ona s leve strane, na razdaljini od preko jednog kilometra.

Now, in both cases, there's still a human in the loop

U oba slučaja još uvek je čovek uključen

to make that lethal firing decision,

da donese odluku o smrtonosnoj paljbi,

but it's not a technological requirement. It's a choice.

ali to nije zahtev tehnologije. To je izbor.

And it's that choice that I want to focus on,

I taj izbor je ono na šta želim da se fokusiram,

because as we migrate lethal decision-making

jer kako prenosimo donošenje smrtonosne odluke

from humans to software,

sa ljudi na softver,

we risk not only taking the humanity out of war,

rizikujemo ne samo to da izbacimo ljudskost iz rata,

but also changing our social landscape entirely,

nego i da potpuno promenimo sliku našeg društva

far from the battlefield.

daleko od bojnog polja.

That's because the way humans resolve conflict

To je zbog toga što način na koji ljudi razrešavaju sukob

shapes our social landscape.

oblikuje sliku našeg društva.

And this has always been the case, throughout history.

Ovo je uvek bio slučaj tokom istorije.

For example, these were state-of-the-art weapons systems

Na primer, ovo su bili moderni sistemi naoružanja

in 1400 A.D.

1400. godine.

Now they were both very expensive to build and maintain,

Oba su bila skupa za izgradnju i održavanje,

but with these you could dominate the populace,

ali njima ste mogli da vladate narodom

and the distribution of political power in feudal society reflected that.

i to se ogledalo u raspodeli političke moći u feudalnom društvu.

Power was focused at the very top.

Moć je bila koncentrisana u samom vrhu.

And what changed? Technological innovation.

Šta se promenilo? Tehnološka inovacija.

Gunpowder, cannon.

Barut, top.

And pretty soon, armor and castles were obsolete,

I uskoro, oklopi i tvrđave su postali beskorisni

and it mattered less who you brought to the battlefield

i bilo je manje važno koga ste doveli na bojno polje

versus how many people you brought to the battlefield.

od toga koliko ste ljudi doveli na bojno polje.

And as armies grew in size, the nation-state arose

I kako su vojske rasle, nastajala je nacionalna država

as a political and logistical requirement of defense.

kao politički i logistički zahtev odbrane.

And as leaders had to rely on more of their populace,

I kako su lideri morali da se oslanjaju više na svoj narod,

they began to share power.

počeli su da dele moć.

Representative government began to form.

Počela je da se formira vlada.

So again, the tools we use to resolve conflict

I opet, sredstva koja koristimo da razrešimo sukob

shape our social landscape.

oblikuju sliku našeg društva.

Autonomous robotic weapons are such a tool,

Autonomna robotička oružja su takvo sredstvo,

except that, by requiring very few people to go to war,

koje osim što zahteva da vrlo malo ljudi pođe u rat,

they risk re-centralizing power into very few hands,

rizikuje ponovno centralizovanje moći u mali broj ruku

possibly reversing a five-century trend toward democracy.

što bi verovatno obrnulo petovekovni trend ka demokratiji.

Now, I think, knowing this,

Znajući ovo, mislim da možemo

we can take decisive steps to preserve our democratic institutions,

preduzeti odlučne korake da sačuvamo naše demokratske institucije,

to do what humans do best, which is adapt.

da uradimo ono što ljudi rade najbolje, a to je da se adaptiramo.

But time is a factor.

Ali vreme je faktor.

Seventy nations are developing remotely-piloted

Sedamdeset nacija razvija sopstvene daljinski upravljane

combat drones of their own,

bespilotne letelice

and as you'll see, remotely-piloted combat drones

i kao što ćete videti, daljinski upravljane bespilotne letelice

are the precursors to autonomous robotic weapons.

su preteče autonomnih robotskih letelica.

That's because once you've deployed remotely-piloted drones,

To je zato što, kad ste razmestili daljinske bespilotne letelice,

there are three powerful factors pushing decision-making

postoje tri moćna faktora koji podstiču premeštanje donošenja odluke

away from humans and on to the weapon platform itself.

sa ljudi na sama oružja.

The first of these is the deluge of video that drones produce.

Prvi od njih je poplava video snimaka koje prave bespilotne letelice.

For example, in 2004, the U.S. drone fleet produced

Na primer 2004., flota bespilotnih letelica SAD-a je proizvela

a grand total of 71 hours of video surveillance for analysis.

ukupno 71 sat video nadzora za analizu.

By 2011, this had gone up to 300,000 hours,

Do 2011., taj broj se popeo na 300.000 sati,

outstripping human ability to review it all,

prevazilazeći ljudsku sposobnost da sve to pregleda

but even that number is about to go up drastically.

i taj broj će se drastično povećavati.

The Pentagon's Gorgon Stare and Argus programs

Programi Pentagona "Gorgonin pogled" i "Argus"

will put up to 65 independently operated camera eyes

će postaviti do 65 nezavisnih objektiva kamere

on each drone platform,

na svaku bespilotnu letelicu

and this would vastly outstrip human ability to review it.

i ovo će uveliko prevazići ljudsku sposobnost da sve to pregleda.

And that means visual intelligence software will need

To znači da će vizuelno inteligentni softver morati

to scan it for items of interest.

da to skenira da bi izdvojio ono što je značajno.

And that means very soon

A to znači da će uskoro

drones will tell humans what to look at,

bespilotne letelice govoriti ljudima šta da pogledaju,

not the other way around.

a ne obratno.

But there's a second powerful incentive pushing

Ali tu je i drugi moćni podsticaj

decision-making away from humans and onto machines,

za prenošenje donošenja odluka sa ljudi na mašine

and that's electromagnetic jamming,

i to je elektromagnetno ometanje

severing the connection between the drone

koje prekida vezu između bespilotne letelice

and its operator.

i onoga ko njom upravlja.

Now we saw an example of this in 2011

Videli smo primer ovoga 2011.

when an American RQ-170 Sentinel drone

kada se američka bespilotna letelica RQ-170 "Sentinel"

got a bit confused over Iran due to a GPS spoofing attack,

malo zbunila iznad Irana usled napada pomoću GPS obmane,

but any remotely-piloted drone is susceptible to this type of attack,

ali bilo koja daljinska bespilotna letelica bila bi podložna ovom tipu napada,

and that means drones

a to znači da će bespilotne letelice

will have to shoulder more decision-making.

morati u većoj meri da preuzmu donošenje odluke.

They'll know their mission objective,

One će znati cilj svoje misije

and they'll react to new circumstances without human guidance.

i reagovaće na nove okolnosti bez ljudskog vođstva.

They'll ignore external radio signals

One će ignorisati spoljne radio signale

and send very few of their own.

i slati vrlo malo sopstvenih.

Which brings us to, really, the third

I to nas dovodi do trećeg

and most powerful incentive pushing decision-making

i najmoćnijeg podsticaja za prenošenje donošenja odluka

away from humans and onto weapons:

sa ljudi na oružje:

plausible deniability.

uverljivo negiranje.

Now we live in a global economy.

Živimo u globalnoj ekonomiji.

High-tech manufacturing is occurring on most continents.

Proizvodnja visoke tehnologije se dešava na uglavnom svim kontinentima.

Cyber espionage is spiriting away advanced designs

Sajber špijunaža krijumčari napredne dizajne

to parts unknown,

na nepoznate destinacije

and in that environment, it is very likely

i u takvom okruženju je vrlo verovatno

that a successful drone design will be knocked off in contract factories,

da će uspešan dizajn bespilotne letelice stati u fabrikama

proliferate in the gray market.

i umnožiti se na crnom tržištu.

And in that situation, sifting through the wreckage

I u toj situaciji, pretraživanjem olupine

of a suicide drone attack, it will be very difficult to say

posle samoubilačkog napada bespilotne letelice, biće vrlo teško reći

who sent that weapon.

ko je poslao to oružje.

This raises the very real possibility

Ovo predstavlja realnu mogućnost

of anonymous war.

anonimnog rata.

This could tilt the geopolitical balance on its head,

Ovo bi moglo okrenuti geopolitičku ravnotežu naglavce

make it very difficult for a nation to turn its firepower

i vrlo otežati naciji da upotrebi svoju oružanu moć

against an attacker, and that could shift the balance

protiv napadača i to može da promeni ravnotežu

in the 21st century away from defense and toward offense.

u 21. veku sa odbrane na napad.

It could make military action a viable option

Može da učini vojnu akciju izvodljivom opcijom

not just for small nations,

ne samo za male narode,

but criminal organizations, private enterprise,

nego i za kriminalne organizacije, privatne poduhvate,

even powerful individuals.

pa čak i za moćne pojedince.

It could create a landscape of rival warlords

Može da kreira sliku u kojoj rivalski gospodari rata

undermining rule of law and civil society.

podrivaju zakon prava i civilnog društva.

Now if responsibility and transparency

Ako su odgovornost i transparentnost

are two of the cornerstones of representative government,

dva stuba vlade,

autonomous robotic weapons could undermine both.

autonomna robotska oružja bi mogla da ih potkopaju.

Now you might be thinking that

Možda mislite da bi građani

citizens of high-tech nations

visoko tehnoloških nacija

would have the advantage in any robotic war,

imali prednost u bilo kom robotskom ratu,

that citizens of those nations would be less vulnerable,

da bi građani ovih nacija bili manje ranjivi,

particularly against developing nations.

posebno u poređenju sa nacijama u razvoju.

But I think the truth is the exact opposite.

Ali mislim da je istina potpuno suprotna.

I think citizens of high-tech societies

Mislim da su građani visoko tehnoloških društava

are more vulnerable to robotic weapons,

ranjiviji za robotska oružja

and the reason can be summed up in one word: data.

i razlog za to može da se sumira u jednoj reči: podaci.

Data powers high-tech societies.

Podaci vladaju visoko tehnološkim društvima.

Cell phone geolocation, telecom metadata,

Geolokacija mobilnih telefona, telekomunikacioni metapodaci,

social media, email, text, financial transaction data,

društveni mediji, e-mail, SMS, podaci finansijskih transakcija

transportation data, it's a wealth of real-time data

saobraćajni podaci, to je bogatstvo podataka u realnom vremenu

on the movements and social interactions of people.

o potezima i društvenim interakcijama ljudi.

In short, we are more visible to machines

Ukratko, vidljiviji smo mašinama

than any people in history,

od bilo kojih drugih ljudi u istoriji

and this perfectly suits the targeting needs of autonomous weapons.

i ovo savršeno odgovara ciljanim potrebama autonomnog oružja.

What you're looking at here

Ono što vidite ovde

is a link analysis map of a social group.

je mapa analize veza društvene grupe.

Lines indicate social connectedness between individuals.

Linije označavaju društvenu povezanost između osoba.

And these types of maps can be automatically generated

Ova vrsta mapa može se generisati automatski

based on the data trail modern people leave behind.

na osnovu tragova podataka koje moderni ljudi ostavljaju za sobom.

Now it's typically used to market goods and services

Obično se koristi u tržištu robe i usluga

to targeted demographics, but it's a dual-use technology,

za ciljanu demografiju, ali to je tehnologija za obe strane

because targeting is used in another context.

jer je "ciljanje" upotrebljeno u drugom kontekstu.

Notice that certain individuals are highlighted.

Vidite da su određene osobe istaknute.

These are the hubs of social networks.

Reč je o centrima društvenih mreža.

These are organizers, opinion-makers, leaders,

To su organizatori, oni utiču na formiranje mišljenja, vođe

and these people also can be automatically identified

i ti ljudi takođe mogu biti automatski identifikovani

from their communication patterns.

pomoću svojih obrazaca komunikacije.

Now, if you're a marketer, you might then target them

Ako ste prodavac, onda ih možete naciljati

with product samples, try to spread your brand

pomoću uzoraka proizvoda i pokušati da proširite svoj brend

through their social group.

u njihovoj društvenoj grupi.

But if you're a repressive government

Ali, ako ste represivna vlada

searching for political enemies, you might instead remove them,

koja traži političke protivnike, mogli biste umesto toga da ih uklonite,

eliminate them, disrupt their social group,

eliminišete, poremetite njihovu društvenu grupu

and those who remain behind lose social cohesion

i one koji stoje u labavoj društvenoj koheziji

and organization.

i organizaciji.

Now in a world of cheap, proliferating robotic weapons,

U svetu jeftinih robotskih oružja koja se sve više umnožavaju

borders would offer very little protection

granice bi pružale vrlo malu zaštitu

to critics of distant governments

kritičarima udaljenih vlada

or trans-national criminal organizations.

ili međunarodnim kriminalnim organizacijama.

Popular movements agitating for change

Narodni pokreti koji agituju za promenu

could be detected early and their leaders eliminated

mogli bi biti otkriveni rano i njihovi lideri eliminisani

before their ideas achieve critical mass.

pre neko što njihove ideje usvoji kritična masa.

And ideas achieving critical mass

A ideje koje dostižu kritičnu masu

is what political activism in popular government is all about.

su ono čime se politički aktivizam u narodnoj vladi bavi.

Anonymous lethal weapons could make lethal action

Anonimna smrtonosna oružja mogu preduzeti smrtonosnu akciju,

an easy choice for all sorts of competing interests.

lak izbor za sve vrste takmičarskih interesa.

And this would put a chill on free speech

I ovo bi stavilo tačku na slobodu govora

and popular political action, the very heart of democracy.

i političku aktivnost naroda, samu srž demokratije.

And this is why we need an international treaty

To je ono zbog čega nam treba međunarodni sporazum

on robotic weapons, and in particular a global ban

o robotskom oružju i posebno globalna zabrana

on the development and deployment of killer robots.

razvoja i razmeštanja robota ubica.

Now we already have international treaties

Već imamo međunarodne sporazume

on nuclear and biological weapons, and, while imperfect,

o nuklearnom i biološkom oružju koji su uglavnom funkcionisali,

these have largely worked.

iako nisu savršeni.

But robotic weapons might be every bit as dangerous,

Ali robotska oružja mogu biti isto tako opasna

because they will almost certainly be used,

jer će ona skoro sigurno biti upotrebljena

and they would also be corrosive to our democratic institutions.

i bila bi isto tako nagrizajuća za naše demokratske institucije.

Now in November 2012 the U.S. Department of Defense

U novembru 2012. Ministarstvo odbrane Sjedinjenih Država

issued a directive requiring

izdalo je direktivu koja zahteva

a human being be present in all lethal decisions.

da čovek bude prisutan kod svih smrtonosnih odluka.

This temporarily effectively banned autonomous weapons in the U.S. military,

Ovim su u stvari zabranjena autonomna oružja u američkoj vojsci,

but that directive needs to be made permanent.

ali ta direktiva bi trebalo da postane trajna.

And it could set the stage for global action.

To bi moglo da pripremi pozornicu za globalnu akciju.

Because we need an international legal framework

Zato što nam treba međunarodni zakonski okvir

for robotic weapons.

za robotska oružja.

And we need it now, before there's a devastating attack

I potreban nam je odmah, pre nego što se desi razoran napad

or a terrorist incident that causes nations of the world

ili teroristički incident koji uzrokuje da nacije sveta

to rush to adopt these weapons

požure da usvoje ova oružja

before thinking through the consequences.

pre nego što razmisle o posledicama.

Autonomous robotic weapons concentrate too much power

Autonomna robotska oružja koncentrišu previše moći

in too few hands, and they would imperil democracy itself.

u suviše malom broju ruku i to bi ugrozilo samu demokratiju.

Now, don't get me wrong, I think there are tons

Nemojte me shvatiti pogrešno, mislim da ima mnogo

of great uses for unarmed civilian drones:

sjajnih načina za upotrebu nenaoružanih civilnih bespilotnih letelica:

environmental monitoring, search and rescue, logistics.

nadgledanje okoline, potraga i spasavanje, logistika.

If we have an international treaty on robotic weapons,

Ako bismo imali međunarodni sporazum o robotskom oružju,

how do we gain the benefits of autonomous drones

kako bismo onda dobili koristi od autonomnih bespilotnih letelica

and vehicles while still protecting ourselves

i vozila, dok se u isto vreme čuvamo

against illegal robotic weapons?

od ilegalnog robotskog oružja?

I think the secret will be transparency.

Mislim da će tajna biti u transparentnosti.

No robot should have an expectation of privacy

Nijedan robot ne bi trebalo da očekuje da ima privatnost

in a public place.

u javnom prostoru.

(Applause)

(Aplauz)

Each robot and drone should have

Svaki robot i bespilotna letelica bi trebalo da ima

a cryptographically signed I.D. burned in at the factory

kriptografsku identifikaciju utisnutu u fabrici

that can be used to track its movement through public spaces.

koja može da se iskoristi za praćenje njegovog kretanja kroz javne prostore.

We have license plates on cars, tail numbers on aircraft.

Imamo tablice na kolima, brojeve na repovima aviona.

This is no different.

Ovo nije ništa drugačije.

And every citizen should be able to download an app

Svaki građanin bi trebalo da može da skine aplikaciju

that shows the population of drones and autonomous vehicles

koja prikazuje populaciju bespilotnih letelica i autonomnih vozila

moving through public spaces around them,

koja se kreće u javnim prostorima oko njih

both right now and historically.

i u sadašnjem trenutku i u prošlosti.

And civic leaders should deploy sensors and civic drones

I gradske vođe bi trebalo da postave senzore i gradske bespilotne letelice

to detect rogue drones,

da otkrivaju nezakonite letelice

and instead of sending killer drones of their own up to shoot them down,

i umesto da šalju same letelice ubice da ih obore,

they should notify humans to their presence.

trebalo bi da obaveste ljude o njihovom prisustvu.

And in certain very high-security areas,

I u određenim oblastima visoke bezbednosti

perhaps civic drones would snare them

možda bi gradske bespilotne letelice mogle da ih uhvate

and drag them off to a bomb disposal facility.

i odvuku ih do mesta za odlaganje bombi.

But notice, this is more an immune system

Ali vidite, ovo je otporniji sistem

than a weapons system.

od sistema oružja.

It would allow us to avail ourselves of the use

To bi nam omogućilo da iskoristimo upotrebu

of autonomous vehicles and drones

autonomnih vozila i bespilotnih letelica

while still preserving our open, civil society.

dok u isto vreme čuvamo naše otvoreno, građansko društvo.

We must ban the deployment and development

Moramo zabraniti razmeštanje i razvoj

of killer robots.

robota ubica.

Let's not succumb to the temptation to automate war.

Nemojmo da podlegnemo iskušenju da automatizujemo rat.

Autocratic governments and criminal organizations

Apsolutističke vlade i kriminalne organizacije

undoubtedly will, but let's not join them.

bez sumnje hoće, ali nemojmo da im se pridružimo.

Autonomous robotic weapons

Autonomna robotska oružja

would concentrate too much power

bi koncentrisala suviše moći

in too few unseen hands,

u suviše mali broj nevidljivih ruku

and that would be corrosive to representative government.

i to bi bilo nagrizajuće za predstavnike vlasti.

Let's make sure, for democracies at least,

Osigurajmo da, barem za demokratije,

killer robots remain fiction.

roboti ubice ostanu fikcija.

Thank you.

Hvala vam.

(Applause)

(Aplauz)

Thank you. (Applause)

Hvala vam. (Aplauz)