What happens when our computers get smarter than we are? | Nick Bostrom

Prevodilac: Miloš Milosavljević Lektor: Mile Živković

I work with a bunch of mathematicians, philosophers and computer scientists,

Ja radim sa grupom matematičara, filozofa i informatičara

and we sit around and think about the future of machine intelligence,

i mi, između ostalog, sedimo i razmišljamo o budućnosti

among other things.

mašinske inteligencije.

Some people think that some of these things are sort of science fiction-y,

Neki misle da su neke od ovih stvari kao naučna fantastika,

far out there, crazy.

uvrnute, lude.

But I like to say,

Ali ja volim da kažem:

okay, let's look at the modern human condition.

pogledajmo kako izgleda moderni čovek.

(Laughter)

(Smeh)

This is the normal way for things to be.

Normalno je da bude ovako.

But if we think about it,

Ali ako razmislimo,

we are actually recently arrived guests on this planet,

mi smo, kao ljudska vrsta, skoro pristigli gosti

the human species.

na ovoj planeti.

Think about if Earth was created one year ago,

Zamislite da je Zemlja nastala pre godinu dana.

the human species, then, would be 10 minutes old.

Ljudska vrsta bi onda bila stara 10 minuta,

The industrial era started two seconds ago.

a industrijsko doba je počelo pre dve sekunde.

Another way to look at this is to think of world GDP over the last 10,000 years,

Drugi način da predstavimo ovo

jeste da pogledamo svetski BDP u poslednjih 10.000 godina.

I've actually taken the trouble to plot this for you in a graph.

Potrudio sam se da vam ovo predstavim grafički.

It looks like this.

To izgleda ovako.

(Laughter)

(Smeh)

It's a curious shape for a normal condition.

U normalnoj situaciji, to je neobičan oblik.

I sure wouldn't want to sit on it.

Sigurno ne bih voleo da sednem na njega.

(Laughter)

(Smeh)

Let's ask ourselves, what is the cause of this current anomaly?

Zapitajmo se: šta je razlog ove savremene anomalije?

Some people would say it's technology.

Neki bi rekli da je to tehnologija.

Now it's true, technology has accumulated through human history,

Istina je, tehnologija se nagomilala tokom ljudske istorije,

and right now, technology advances extremely rapidly --

a sada napreduje ekstremno brzo.

that is the proximate cause,

To je neposredan uzrok,

that's why we are currently so very productive.

zbog toga smo toliko produktivni.

But I like to think back further to the ultimate cause.

Ali želeo bih da se vratim unazad do prvobitnog uzroka.

Look at these two highly distinguished gentlemen:

Pogledajte ova dva cenjena gospodina.

We have Kanzi --

Imamo Kanzija -

he's mastered 200 lexical tokens, an incredible feat.

on je ovladao sa 200 leksičkih simbola, što je neverovatan podvig.

And Ed Witten unleashed the second superstring revolution.

A Ed Viten je pokrenuo drugu revoluciju superstruna.

If we look under the hood, this is what we find:

Ako pogledamo ispod haube, pronaći ćemo ovo:

basically the same thing.

u osnovi istu stvar.

One is a little larger,

Jedna je malo veća

it maybe also has a few tricks in the exact way it's wired.

i možda ima par trikova u načinu na koji je povezana.

These invisible differences cannot be too complicated, however,

Ove nevidljive razlike ne mogu biti previše komplikovane,

because there have only been 250,000 generations

jer je prošlo samo 250.000 generacija

since our last common ancestor.

od našeg poslednjeg zajedničkog pretka.

We know that complicated mechanisms take a long time to evolve.

Znamo da komplikovanim mehanizmima treba mnogo vremena da se razviju.

So a bunch of relatively minor changes

Tako da skup relativno malih promena

take us from Kanzi to Witten,

nas od Kanzija dovodi do Vitena,

from broken-off tree branches to intercontinental ballistic missiles.

od slomljenih grana drveća, do interkontinentalnih raketa.

So this then seems pretty obvious that everything we've achieved,

Onda izgleda prilično očigledno da sve što smo postigli

and everything we care about,

i sve što nam je važno

depends crucially on some relatively minor changes that made the human mind.

zavisi od nekih relativno malih promena koje su stvorile ljudski um.

And the corollary, of course, is that any further changes

I onda, logično, sve dalje promene

that could significantly change the substrate of thinking

koje značajno menjaju osnovu mišljenja

could have potentially enormous consequences.

mogle bi imati potencijalno ogromne posledice.

Some of my colleagues think we're on the verge

Neke moje kolege misle da se nalazimo nadomak

of something that could cause a profound change in that substrate,

nečega što bi moglo da prouzrokuje duboku promenu u toj osnovi,

and that is machine superintelligence.

a to je mašinska superinteligencija.

Artificial intelligence used to be about putting commands in a box.

Veštačka inteligencija je dosad bila kutija u koju ste ubacivali komande.

You would have human programmers

Ljudski programeri

that would painstakingly handcraft knowledge items.

su brižljivo brusili kolekcije znanja.

You build up these expert systems,

Razvijali su te ekspertske sisteme

and they were kind of useful for some purposes,

i oni su bili korisni za neke namene,

but they were very brittle, you couldn't scale them.

ali su bili vrlo krhki, niste mogli da ih podešavate.

Basically, you got out only what you put in.

U suštini, dobijali ste samo ono što ste već imali ubačeno.

But since then,

Ali od tada,

a paradigm shift has taken place in the field of artificial intelligence.

desila se promena paradigme na polju veštačke inteligencije.

Today, the action is really around machine learning.

Danas se najviše radi na mašinskom učenju.

So rather than handcrafting knowledge representations and features,

Ne kreiramo imitacije znanja i njegove komponente,

we create algorithms that learn, often from raw perceptual data.

već kreiramo algoritme koji uče iz sirovih opažajnih podataka.

Basically the same thing that the human infant does.

U suštini, isto kao što to radi novorođenče.

The result is A.I. that is not limited to one domain --

Rezultat je veštačka inteligencija (V.I.) koja nije ograničena na jednu oblast -

the same system can learn to translate between any pairs of languages,

isti sistem može da nauči da prevodi između bilo koja dva jezika

or learn to play any computer game on the Atari console.

ili da nauči da igra bilo koju igricu na Atariju.

Now of course,

Naravno,

A.I. is still nowhere near having the same powerful, cross-domain

V.I. je još uvek daleko od toga da može kao i ljudi

ability to learn and plan as a human being has.

da uči i planira između više oblasti istovremeno.

The cortex still has some algorithmic tricks

Korteks ima neke algoritamske trikove

that we don't yet know how to match in machines.

koje još uvek ne znamo kako da postignemo kod mašina.

So the question is,

Tako da, pitanje je:

how far are we from being able to match those tricks?

koliko smo daleko od toga da to postignemo?

A couple of years ago,

Pre nekoliko godina

we did a survey of some of the world's leading A.I. experts,

anketirali smo neke vodeće stručnjake za V.I.

to see what they think, and one of the questions we asked was,

da vidimo šta oni misle i jedno od naših pitanja je bilo:

"By which year do you think there is a 50 percent probability

"Do koje godine mislite da postoji 50% šanse

that we will have achieved human-level machine intelligence?"

da ćemo kod mašina postići inteligenciju koja je na nivou ljudske?"

We defined human-level here as the ability to perform

Definisali smo "ljudski nivo" kao sposobnost da se obavi

almost any job at least as well as an adult human,

skoro svaki posao onako kao što bi to uradila odrasla osoba.

so real human-level, not just within some limited domain.

Znači, na pravom ljudskom nivou, a ne u nekoj ograničenoj oblasti.

And the median answer was 2040 or 2050,

I srednji odgovor je bio: oko 2040. ili 2050,

depending on precisely which group of experts we asked.

u zavisnosti od toga koju smo tačno grupu stručnjaka pitali.

Now, it could happen much, much later, or sooner,

Moglo bi da se desi i mnogo, mnogo kasnije, ili ranije.

the truth is nobody really knows.

U stvari, niko to ne zna.

What we do know is that the ultimate limit to information processing

Ono što znamo je da se konačna granica obrade informacija

in a machine substrate lies far outside the limits in biological tissue.

kod mašina nalazi daleko izvan granica biološkog tkiva.

This comes down to physics.

Ovo se svodi na fiziku.

A biological neuron fires, maybe, at 200 hertz, 200 times a second.

Biološki neuron radi, možda, na 200 herca, 200 puta u sekundi.

But even a present-day transistor operates at the Gigahertz.

Ali čak i današnji tranzistor radi na jednom gigahercu.

Neurons propagate slowly in axons, 100 meters per second, tops.

Neuroni se razmnožavaju sporo u aksonima - najviše 100 m u sekundi.

But in computers, signals can travel at the speed of light.

Ali u kompjuteru, signali mogu da putuju brzinom svetlosti.

There are also size limitations,

Takođe postoje ograničenja u veličini,

like a human brain has to fit inside a cranium,

kao što ih ima ljudski mozak da bi stao u lobanju,

but a computer can be the size of a warehouse or larger.

ali kompjuter može biti veličine skladišta ili veći.

So the potential for superintelligence lies dormant in matter,

Tako da potencijal superinteligencije čeka uspavan,

much like the power of the atom lay dormant throughout human history,

kao što je moć atoma bila uspavana tokom ljudske istorije,

patiently waiting there until 1945.

strpljivo čekajući do 1945.

In this century,

U ovom veku,

scientists may learn to awaken the power of artificial intelligence.

naučnici možda nauče da probude moć veštačke inteligencije.

And I think we might then see an intelligence explosion.

I mislim da bismo tada mogli da vidimo eksploziju inteligencije.

Now most people, when they think about what is smart and what is dumb,

Većina ljudi kad pomisli šta je pametno, a šta glupo,

I think have in mind a picture roughly like this.

mislim da u glavi ima sliku sličnu ovoj.

So at one end we have the village idiot,

Na jednom kraju imamo idiota,

and then far over at the other side

a daleko na drugom kraju

we have Ed Witten, or Albert Einstein, or whoever your favorite guru is.

imamo Eda Vitena, ili Alberta Ajnštajna, ili nekog vašeg omiljenog gurua.

But I think that from the point of view of artificial intelligence,

Ali s tačke gledišta veštačke inteligencije,

the true picture is actually probably more like this:

prava slika više izgleda ovako:

AI starts out at this point here, at zero intelligence,

V.i. počinje u ovoj ovde tački, na nultoj inteligenciji

and then, after many, many years of really hard work,

i onda, posle mnogo, mnogo godina vrednog rada,

maybe eventually we get to mouse-level artificial intelligence,

možda konačno stignemo do veštačke inteligencije na nivou miša,

something that can navigate cluttered environments

koja može da se kreće u zakrčenim prostorima

as well as a mouse can.

kao što to može miš.

And then, after many, many more years of really hard work, lots of investment,

I zatim, posle još mnogo godina vrednog rada i mnogo ulaganja,

maybe eventually we get to chimpanzee-level artificial intelligence.

možda konačno stignemo do nivoa šimpanze,

And then, after even more years of really, really hard work,

a posle još više godina vrednog rada,

we get to village idiot artificial intelligence.

do nivoa veštačke inteligencije idiota.

And a few moments later, we are beyond Ed Witten.

A nekoliko trenutaka kasnije, pretekli smo Eda Vitena.

The train doesn't stop at Humanville Station.

Voz se neće zaustaviti kad pretekne ljude,

It's likely, rather, to swoosh right by.

već će najverovatnije, samo prošišati pored.

Now this has profound implications,

Ovo ima suštinske posledice,

particularly when it comes to questions of power.

naročito što se tiče pitanja moći.

For example, chimpanzees are strong --

Na primer, šimpanze su snažne -

pound for pound, a chimpanzee is about twice as strong as a fit human male.

šimpanza je oko dva puta snažnija od čoveka.

And yet, the fate of Kanzi and his pals depends a lot more

Pa ipak, sudbina Kanzija i njegovih drugara zavisi više

on what we humans do than on what the chimpanzees do themselves.

od toga šta mi ljudi radimo, nego od toga šta šimpanze rade.

Once there is superintelligence,

Kad se pojavi superinteligencija,

the fate of humanity may depend on what the superintelligence does.

sudbina čovečanstva može zavisiti od toga šta superinteligencija radi.

Think about it:

Razmislite:

Machine intelligence is the last invention that humanity will ever need to make.

mašinska inteligencija je poslednji izum koji će čovečanstvo morati da napravi.

Machines will then be better at inventing than we are,

Mašine će tada biti bolje u pronalazaštvu od nas

and they'll be doing so on digital timescales.

i to će raditi u digitalnim vremenskim rokovima.

What this means is basically a telescoping of the future.

To u suštini znači skraćivanje budućnosti.

Think of all the crazy technologies that you could have imagined

Pomislite na svu neverovatnu tehnologiju koju možete da zamislite

maybe humans could have developed in the fullness of time:

da bi ljudi mogli da razviju kroz vreme:

cures for aging, space colonization,

lek protiv starenja, kolonizaciju svemira,

self-replicating nanobots or uploading of minds into computers,

samoreplikaciju nanorobota ili čuvanje umova u kompjuterima,

all kinds of science fiction-y stuff

raznorazne stvari iz naučne fantastike

that's nevertheless consistent with the laws of physics.

koje su ipak u skladu sa zakonima fizike.

All of this superintelligence could develop, and possibly quite rapidly.

Sve to bi superinteligencija mogla da napravi, i to verovatno jako brzo.

Now, a superintelligence with such technological maturity

Superinteligecija s takvom tehnološkom zrelošću

would be extremely powerful,

bila bi izuzetno moćna

and at least in some scenarios, it would be able to get what it wants.

i, barem po nekim scenarijima, mogla bi da dobije šta hoće.

We would then have a future that would be shaped by the preferences of this A.I.

Onda bismo imali budućnost koja bi bila oblikovana po želji te V.I.

Now a good question is, what are those preferences?

Sada je pitanje: kakve su te želje?

Here it gets trickier.

Ovde nastaje začkoljica.

To make any headway with this,

Da bismo napredovali dalje,

we must first of all avoid anthropomorphizing.

moramo prvo da izbegnemo antropomorfnost.

And this is ironic because every newspaper article

To je ironično, jer svaki novinski članak

about the future of A.I. has a picture of this:

o budućnosti V.I. zamišlja ovo:

So I think what we need to do is to conceive of the issue more abstractly,

Mislim da bi trebalo da zamislimo problem apstraktnije,

not in terms of vivid Hollywood scenarios.

a ne kao živopisni holivudski scenario.

We need to think of intelligence as an optimization process,

Moramo da posmatramo inteligenciju kao proces optimizacije,

a process that steers the future into a particular set of configurations.

proces koji vodi budućnost u određeni set konfiguracija.

A superintelligence is a really strong optimization process.

Superinteligencija je vrlo snažan proces optimizacije.

It's extremely good at using available means to achieve a state

Izuzetno je dobra u korišćenju dostupnih sredstava da postigne stanje

in which its goal is realized.

u kome je njen cilj ostvaren.

This means that there is no necessary connection between

To znači da ne postoji obavezno veza između:

being highly intelligent in this sense,

biti visoko inteligentan u ovom smislu

and having an objective that we humans would find worthwhile or meaningful.

i imati cilj koji mi ljudi smatramo važnim ili smislenim.

Suppose we give an A.I. the goal to make humans smile.

Pretpostavimo da damo V.I. zadatak da nasmeje ljude.

When the A.I. is weak, it performs useful or amusing actions

Kada je V.I. slaba, ona izvodi korisne ili zabavne radnje

that cause its user to smile.

koje prouzrokuju osmeh kod korisnika.

When the A.I. becomes superintelligent,

Kada V.I. postane superinteligentna,

it realizes that there is a more effective way to achieve this goal:

ona shvata da postoji efikasniji način da postigne taj cilj,

take control of the world

a to je da preuzme kontrolu nad svetom

and stick electrodes into the facial muscles of humans

i prikači elektrode na mišiće ljudskih lica

to cause constant, beaming grins.

da bi proizvela konstantne kezove.

Another example,

Drugi primer je:

suppose we give A.I. the goal to solve a difficult mathematical problem.

pretpostavimo da V. I. damo zadatak da reši težak matetematički problem.

When the A.I. becomes superintelligent,

Kad V.I. postane superinteligentna,

it realizes that the most effective way to get the solution to this problem

shvati da je najefikasniji način za rešenje ovog problema

is by transforming the planet into a giant computer,

da transformiše planetu u džinovski kompjuter,

so as to increase its thinking capacity.

kako bi uvećala svoj kapacitet za razmišljanje.

And notice that this gives the A.I.s an instrumental reason

To daje veštačkoj inteligenciji instrumentalni razlog

to do things to us that we might not approve of.

da nam radi stvari s kojima se možda ne slažemo.

Human beings in this model are threats,

Ljudska bića su pretnja u ovom modelu,

we could prevent the mathematical problem from being solved.

mogli bi da budu prepreka za rešavanje matematičkog problema.

Of course, perceivably things won't go wrong in these particular ways;

Naravno, stvari neće krenuti loše baš na ovaj način;

these are cartoon examples.

ovo su primeri iz crtanog filma.

But the general point here is important:

Ali ovde je važna poenta:

if you create a really powerful optimization process

ako kreirate veoma moćan proces optimizacije

to maximize for objective x,

da se postigne maksimalan učinak za cilj X,

you better make sure that your definition of x

onda neka vaša definicija X-a

incorporates everything you care about.

obavezno uključi sve ono do čega vam je stalo.

This is a lesson that's also taught in many a myth.

Ovo je lekcija kojoj nas takođe uče i mnogi mitovi.

King Midas wishes that everything he touches be turned into gold.

Kralj Mida poželi da sve što dodirne bude pretvoreno u zlato.

He touches his daughter, she turns into gold.

I tako, dodirne ćerku i ona se pretvori u zlato.

He touches his food, it turns into gold.

Dodirne hranu i ona se pretvori u zlato.

This could become practically relevant,

To može biti relevantno i u praksi,

not just as a metaphor for greed,

ne samo kao metafora za pohlepu,

but as an illustration of what happens

već i kao ilustracija za ono što bi se desilo

if you create a powerful optimization process

ako pogrešno kreirate moćan proces optimizacije

and give it misconceived or poorly specified goals.

ili loše formulišete ciljeve.

Now you might say, if a computer starts sticking electrodes into people's faces,

Mogli biste reći: pa ako kompjuter počne da kači elektrode ljudima na lica,

we'd just shut it off.

prosto ćemo ga isključiti.

A, this is not necessarily so easy to do if we've grown dependent on the system --

Pod A: to možda ne bi bilo tako lako, ako bismo bili zavisni od sistema -

like, where is the off switch to the Internet?

na primer: gde je prekidač za gašenje interneta?

B, why haven't the chimpanzees flicked the off switch to humanity,

Pod B: zašto šimpanze nisu isključile prekidač čovečanstvu,

or the Neanderthals?

ili Neandertalci?

They certainly had reasons.

Sigurno ima razloga.

We have an off switch, for example, right here.

Imamo prekidač za gašenje: na primer, ovde.

(Choking)

(Gušenje)

The reason is that we are an intelligent adversary;

Razlog je to što smo mi inteligentan neprijatelj;

we can anticipate threats and plan around them.

možemo predvideti pretnje i planirati kako da ih izbegnemo.

But so could a superintelligent agent,

Ali to može i predstavnik superinteligencije,

and it would be much better at that than we are.

i bio bi mnogo bolji u tome od nas.

The point is, we should not be confident that we have this under control here.

Poenta je da ne treba da budemo tako sigurni da imamo kontrolu ovde.

And we could try to make our job a little bit easier by, say,

Mogli bismo da pokušamo da malo olakšamo posao

tako što ćemo, recimo, staviti V.I. u kutiju,

putting the A.I. in a box,

like a secure software environment,

kao bezbedno okruženje za softver,

a virtual reality simulation from which it cannot escape.

simulaciju virtuelne realnosti iz koje ona ne može pobeći.

But how confident can we be that the A.I. couldn't find a bug.

Ali koliko možemo biti sigurni da V.I. neće naći rupu?

Given that merely human hackers find bugs all the time,

S obzirom da ljudski hakeri stalno nalaze rupe,

I'd say, probably not very confident.

ja ne bih bio tako siguran.

So we disconnect the ethernet cable to create an air gap,

Isključimo eternet kabl da bismo stvorili vazdušni zid,

but again, like merely human hackers

ali, kao obični ljudski hakeri,

routinely transgress air gaps using social engineering.

rutinski preskačemo vazdušne zidove pomoću socijalnog inženjeringa.

Right now, as I speak,

Upravo sada, dok govorim,

I'm sure there is some employee out there somewhere

siguran sam da negde postoji neki zaposleni

who has been talked into handing out her account details

koga neko nagovara da mu da podatke o svom računu,

by somebody claiming to be from the I.T. department.

neko ko se predstavlja da je iz IT sektora.

More creative scenarios are also possible,

Ima još mogućih kreativnih scenarija,

like if you're the A.I.,

kao na primer, ako ste vi V.I.,

you can imagine wiggling electrodes around in your internal circuitry

možete zamisliti elektrode u vašem unutrašnjem sistemu

to create radio waves that you can use to communicate.

koje stvaraju radio talase koje možete koristiti za komunikaciju.

Or maybe you could pretend to malfunction,

Ili možete da se pretvarate da ste pokvareni

and then when the programmers open you up to see what went wrong with you,

i kada vas programeri otvore da vide šta nije u redu,

they look at the source code -- Bam! --

vide izvorni kod

the manipulation can take place.

i manipulacija može da se desi.

Or it could output the blueprint to a really nifty technology,

Ili može da prenese šemu na pogodan tehnološki uređaj

and when we implement it,

i kada je sprovedemo,

it has some surreptitious side effect that the A.I. had planned.

ima podrivajući efekat koji je V.I. planirala.

The point here is that we should not be confident in our ability

Poenta je da treba da budemo sigurni

to keep a superintelligent genie locked up in its bottle forever.

da ćemo držati superinteligentnog duha zaključanog u boci zauvek.

Sooner or later, it will out.

Pre ili kasnije će izaći.

I believe that the answer here is to figure out

Mislim da je odgovor to da shvatimo

how to create superintelligent A.I. such that even if -- when -- it escapes,

kako da napravimo V.I. tako da, čak i ako pobegne

it is still safe because it is fundamentally on our side

budemo bezbedni jer je suštinski na našoj strani

because it shares our values.

jer deli naše vrednosti.

I see no way around this difficult problem.

Ne vidim drugo rešenje za ovaj težak problem.

Now, I'm actually fairly optimistic that this problem can be solved.

Ali sam optimista da ovaj problem može da se reši.

We wouldn't have to write down a long list of everything we care about,

Ne bismo morali da pišemo dugu listu svega onoga do čega nam je stalo

or worse yet, spell it out in some computer language

ili, još gore, pišemo to na nekom kompjuterskom jeziku

like C++ or Python,

kao što je C++ ili Pajton,

that would be a task beyond hopeless.

što bi bio beznadežan zadatak.

Instead, we would create an A.I. that uses its intelligence

Umesto toga, napravili bismo V.I. koja koristi svoju inteligenciju

to learn what we value,

da nauči šta mi vrednujemo

and its motivation system is constructed in such a way that it is motivated

i čiji motivacioni sistem je konstruisan tako da je motivisan

to pursue our values or to perform actions that it predicts we would approve of.

da sledi naše vrednosti ili obavlja radnje za koje predviđa da ih mi odobravamo.

We would thus leverage its intelligence as much as possible

Tako bismo iskoristili njenu inteligenciju što je više moguće

to solve the problem of value-loading.

da rešimo problem prenošenja vrednosti.

This can happen,

Ovo može da se desi

and the outcome could be very good for humanity.

i ishod bi bio veoma dobar za čovečanstvo.

But it doesn't happen automatically.

Ali to se ne dešava automatski.

The initial conditions for the intelligence explosion

Početni uslovi za eksploziju inteligencije

might need to be set up in just the right way

možda treba da budu postavljeni na tačno određen način

if we are to have a controlled detonation.

ako želimo da imamo kontrolisanu detonaciju.

The values that the A.I. has need to match ours,

Vrednosti V.I. tribe da se poklope sa našim,

not just in the familiar context,

ne samo u poznatim kontekstima,

like where we can easily check how the A.I. behaves,

gde možemo lako proveriti kako se V.I. ponaša,

but also in all novel contexts that the A.I. might encounter

nego i u novim kontekstima u kojima se V.I. može naći

in the indefinite future.

u neodređenoj budućnosti.

And there are also some esoteric issues that would need to be solved, sorted out:

Postoje i neka ezoterična pitanja koja bi trebalo rešiti:

the exact details of its decision theory,

tačne detalje njene teorije odlučivanja,

how to deal with logical uncertainty and so forth.

kako rešiti logičku nesigurnost itd.

So the technical problems that need to be solved to make this work

Tako da tehnički problemi koje bi trebalo rešiti da bi ovo funkcionisalo

look quite difficult --

izgledaju prilično teški -

not as difficult as making a superintelligent A.I.,

ne toliko kao pravljenje superinteligentne V.I.,

but fairly difficult.

ali prilično teški.

Here is the worry:

Problem je u sledećem:

Making superintelligent A.I. is a really hard challenge.

napraviti superinteligentnu V.I. je vrlo težak izazov.

Making superintelligent A.I. that is safe

Pravljenje superinteligentne V.I. koja je bezbedna

involves some additional challenge on top of that.

uključuje još neke dodatne teškoće.

The risk is that if somebody figures out how to crack the first challenge

Postoji rizik da neko otkrije kako da razbije prvu teškoću,

without also having cracked the additional challenge

a da nije razbio i dodatnu teškoću

of ensuring perfect safety.

obezbeđivanja savršene bezbednosti.

So I think that we should work out a solution

Mislim da bi trebalo da nađemo rešenje

to the control problem in advance,

za kontrolu problema unapred

so that we have it available by the time it is needed.

da bismo ga imali na raspolaganju kad dođe vreme.

Now it might be that we cannot solve the entire control problem in advance

Ali možda ne možemo da rešimo ceo problem kontrole unapred,

because maybe some elements can only be put in place

jer možda neki elementi mogu da se ubace

once you know the details of the architecture where it will be implemented.

tek kad znamo detaljnu strukturu na kojoj bi bilo primenjeno.

But the more of the control problem that we solve in advance,

Ali što veći deo problema kontrole rešimo unapred,

the better the odds that the transition to the machine intelligence era

to su bolje šanse da će tranzicija ka dobu mašinske inteligencije

will go well.

proteći dobro

This to me looks like a thing that is well worth doing

To mi izgleda kao stvar koju bi vredelo uraditi

and I can imagine that if things turn out okay,

i mogu da zamislim da ako stvari ispadnu okej,

that people a million years from now look back at this century

ljudi će za milion godina pogledati unazad na ovo doba

and it might well be that they say that the one thing we did that really mattered

i možda će reći da je ono što je bilo zaista važno

was to get this thing right.

bilo to da smo ovo uradili dobro.

Thank you.

Hvala vam.

(Aplauz)

(Applause)