What happens when our computers get smarter than we are? | Nick Bostrom

Prevoditelj: Lidija Šimunović Recezent: Ivan Stamenković

I work with a bunch of mathematicians, philosophers and computer scientists,

Radim s mnogo matematičara, filozofa i računalnih znanstvenika

and we sit around and think about the future of machine intelligence,

s kojima razmišljam o budućnosti strojne inteligencije,

among other things.

među drugim stvarima.

Some people think that some of these things are sort of science fiction-y,

Neki ljudi misle da su te stvari u zoni znanstvene fantastike,

far out there, crazy.

nešto ludo, nestvarno.

But I like to say,

Ali ja volim reći

okay, let's look at the modern human condition.

ok, pogledajmo uvjete modernoga života.

(Laughter)

(Smijeh)

This is the normal way for things to be.

Ovo je normalno stanje stvari.

But if we think about it,

Ali ako malo razmislimo o tome,

we are actually recently arrived guests on this planet,

mi smo zapravo tek nedavno stigli na ovaj planet,

the human species.

mi, ljudska vrsta.

Think about if Earth was created one year ago,

Kad bismo gledali na Zemlju kao da je stvorena prije godinu dana,

the human species, then, would be 10 minutes old.

onda bi ljudska vrsta bila stara deset minuta.

The industrial era started two seconds ago.

Industrijska je era počela prije dvije sekunde.

Another way to look at this is to think of world GDP over the last 10,000 years,

Drugi pogled na ovo jest taj da promatramo svjetski BDP u zadnjih 10 000 godina.

I've actually taken the trouble to plot this for you in a graph.

Odvojio sam nešto vremena i to za vas grafički prikazao.

It looks like this.

Izgleda ovako.

(Laughter)

(Smijeh)

It's a curious shape for a normal condition.

Neobičan je to oblik za uobičajene uvjete.

I sure wouldn't want to sit on it.

Ja sigurno ne bih sjedio na njemu.

(Laughter)

(Smijeh)

Let's ask ourselves, what is the cause of this current anomaly?

Zapitajmo se koji je uzrok ove trenutne anomalije.

Some people would say it's technology.

Neki bi ljudi rekli da je tehnologija.

Now it's true, technology has accumulated through human history,

Istina je da se tijekom povijesti tehnologija nagomilavala

and right now, technology advances extremely rapidly --

i u ovom se trenutku tehnologija jako brzo razvija --

that is the proximate cause,

to je neposredni uzrok,

that's why we are currently so very productive.

i zbog toga smo trenutno jako produktivni.

But I like to think back further to the ultimate cause.

Ali razmislimo o krajnjem uzroku.

Look at these two highly distinguished gentlemen:

Pogledajte ova dva vrlo istaknuta gospodina:

We have Kanzi --

imamo Kanzija --

he's mastered 200 lexical tokens, an incredible feat.

koji je svladao 200 leksičkih znakova što je nevjerojatan podvig.

And Ed Witten unleashed the second superstring revolution.

I Ed Witten koji je započeo drugu revoluciju superstruna.

If we look under the hood, this is what we find:

Ako pomno pogledamo, uočit ćemo ovo:

basically the same thing.

u principu istu stvar.

One is a little larger,

Jedna je malo veća,

it maybe also has a few tricks in the exact way it's wired.

možda ima par specifičnosti u načinu na koji je sklopljena.

These invisible differences cannot be too complicated, however,

Međutim, ove nevidljive razlike ne mogu biti jako složene

because there have only been 250,000 generations

jer nas dijeli samo 250 000 generacija

since our last common ancestor.

od zadnjeg zajedničkog pretka.

We know that complicated mechanisms take a long time to evolve.

Znamo da se složeni mehanizmi dugo razvijaju.

So a bunch of relatively minor changes

Tako da nas hrpa relativno malih promjena

take us from Kanzi to Witten,

vodi od Kanzija do Wittena,

from broken-off tree branches to intercontinental ballistic missiles.

od polomljenih grana do interkontinentalnih balističkih raketa.

So this then seems pretty obvious that everything we've achieved,

Onda je poprilično očito da sve što smo postigli

and everything we care about,

i sve do čega nam je stalo

depends crucially on some relatively minor changes that made the human mind.

uvelike ovisi o relativno malim promjenama koje čine ljudski um.

And the corollary, of course, is that any further changes

Nužna je posljedica ta da sve daljnje izmjene

that could significantly change the substrate of thinking

koje bi mogle značajno promijeniti način razmišljanja

could have potentially enormous consequences.

mogu imati potencijalno ogromne posljedice.

Some of my colleagues think we're on the verge

Neki od mojih kolega smatraju da smo na rubu

of something that could cause a profound change in that substrate,

nečega što bi moglo izazvati takve temeljite promjene,

and that is machine superintelligence.

a radi se o strojnoj superinteligenciji.

Artificial intelligence used to be about putting commands in a box.

Prije je umjetna inteligencija bila pisanje naredbi u kutiju.

You would have human programmers

Ljudski bi programeri

that would painstakingly handcraft knowledge items.

mukotrpno ručno stvarali stavke znanja.

You build up these expert systems,

Izgradite takve stručne sustave

and they were kind of useful for some purposes,

koji su za određene svrhe korisni,

but they were very brittle, you couldn't scale them.

ali su bili krhki i nisu bili mjerljivi.

Basically, you got out only what you put in.

U principu ste dobivali samo ono što ste unijeli.

But since then,

Ali, od onda se

a paradigm shift has taken place in the field of artificial intelligence.

dogodila promjena u pristupu u polju umjetne inteligencije.

Today, the action is really around machine learning.

Danas se uglavnom radi o učenju strojeva.

So rather than handcrafting knowledge representations and features,

Umjesto ručnog unosa prikaza znanja i značajki,

we create algorithms that learn, often from raw perceptual data.

stvaramo algoritme koji uče, i to često iz čistih perceptualnih podataka.

Basically the same thing that the human infant does.

Načelno je to ista stvar koju čini ljudsko dijete.

The result is A.I. that is not limited to one domain --

Rezultat je taj da UI nije ograničena na jedno područje --

the same system can learn to translate between any pairs of languages,

isti sustav može naučiti prevoditi bilo koju kombinaciju jezika

or learn to play any computer game on the Atari console.

ili naučiti igrati bilo koju računalnu igru na Atariju.

Now of course,

Naravno,

A.I. is still nowhere near having the same powerful, cross-domain

UI nije ni blizu posjedovanja iste jake, unakrsne sposobnosti

ability to learn and plan as a human being has.

učenja i planiranja raznih područja kao što to radi ljudsko biće.

The cortex still has some algorithmic tricks

Korteks ima neke algoritamske trikove

that we don't yet know how to match in machines.

koje ne znamo kako prenijeti strojevima.

So the question is,

Tako da je pitanje

how far are we from being able to match those tricks?

koliko smo blizu tome da uskladimo te trikove.

A couple of years ago,

Prije nekoliko godina

we did a survey of some of the world's leading A.I. experts,

proveli smo istraživanje s vodećim stručnjacima za UI

to see what they think, and one of the questions we asked was,

kako bismo vidjeli što misle, i jedno od pitanja bilo je

"By which year do you think there is a 50 percent probability

"Što mislite, do koje će godine biti 50% vjerojatnosti

that we will have achieved human-level machine intelligence?"

da ćemo postići strojnu inteligenciju na razini ljudske inteligencije?"

We defined human-level here as the ability to perform

Ljudsku smo razinu definirali kao sposobnost da se bilo koji zadatak

almost any job at least as well as an adult human,

obavi jednako dobro kao što bi ga obavio odrastao čovjek,

so real human-level, not just within some limited domain.

znači, stvarna ljudska razina, a ne samo unutar određenog područja.

And the median answer was 2040 or 2050,

Srednji je odgovor bio 2040. ili 2050.,

depending on precisely which group of experts we asked.

ovisno o skupini stručnjaka kojoj smo pitanje postavili.

Now, it could happen much, much later, or sooner,

To bi se moglo dogoditi puno, puno kasnije ili ranije,

the truth is nobody really knows.

istina je ta da nitko to zapravo ne zna.

What we do know is that the ultimate limit to information processing

Ali ono što znamo jest to da krajnja granica procesuiranja informacija

in a machine substrate lies far outside the limits in biological tissue.

u strojevima uvelike nadilazi ograničenja biološkog tkiva.

This comes down to physics.

Radi se o fizici.

A biological neuron fires, maybe, at 200 hertz, 200 times a second.

Biološki se neuron ispaljuje na možda 200 herca, 200 puta u sekundi.

But even a present-day transistor operates at the Gigahertz.

Ali čak i današnji tranzistor radi u gigahercima.

Neurons propagate slowly in axons, 100 meters per second, tops.

Neuroni se polako šire u aksone, najviše 100 metara u sekundi.

But in computers, signals can travel at the speed of light.

Ali u računalima signali mogu putovati brzinom svjetlosti.

There are also size limitations,

I tamo postoje ograničenja u veličini,

like a human brain has to fit inside a cranium,

isto kao što ljudski mozak mora stati u lubanju,

but a computer can be the size of a warehouse or larger.

ali računalo može biti veličine skladišta, ili čak veće.

So the potential for superintelligence lies dormant in matter,

Potencijal superinteligencije prikriven je i čeka,

much like the power of the atom lay dormant throughout human history,

slično kao što je snaga atoma bila prikrivena tijekom povijesti,

patiently waiting there until 1945.

strpljivo čekajući 1945. godinu.

In this century,

U ovom su stoljeću

scientists may learn to awaken the power of artificial intelligence.

znanstvenici možda otkrili kako probuditi moć umjetne inteligencije.

And I think we might then see an intelligence explosion.

A onda bismo mogli svjedočiti eksploziji inteligencije.

Now most people, when they think about what is smart and what is dumb,

Većina ljudi, kada razmišlja o tome što je pametno, a što glupo,

I think have in mind a picture roughly like this.

mislim da otprilike zamišlja ovako nešto.

So at one end we have the village idiot,

Na jednom kraju imamo seosku budalu,

and then far over at the other side

a na drugome

we have Ed Witten, or Albert Einstein, or whoever your favorite guru is.

Eda Wittena ili Alberta Einsteina, ili već preferiranog gurua.

But I think that from the point of view of artificial intelligence,

Ali iz perspektive umjetne inteligencije,

the true picture is actually probably more like this:

ta slika vjerojatno izgleda ovako:

AI starts out at this point here, at zero intelligence,

sve počinje na točki nulte inteligencije,

and then, after many, many years of really hard work,

a onda nakon puno, puno godina napornog rada

maybe eventually we get to mouse-level artificial intelligence,

možda konačno dođemo do umjetne inteligencije na razini miša,

something that can navigate cluttered environments

nešto što se može kretati kroz pretrpan prostor

as well as a mouse can.

jednako dobro kao miševi.

And then, after many, many more years of really hard work, lots of investment,

A onda nakon još puno, puno godina napornog rada i mnogo ulaganja,

maybe eventually we get to chimpanzee-level artificial intelligence.

možda dođemo do umjetne inteligencije na razini čimpanze.

And then, after even more years of really, really hard work,

A onda, nakon još puno godina vrlo, vrlo napornog rada,

we get to village idiot artificial intelligence.

dođemo do seoske budale umjetne inteligencije.

And a few moments later, we are beyond Ed Witten.

I nakon nekoliko trenutaka nadiđemo Eda Wittena.

The train doesn't stop at Humanville Station.

Tu napredak ne prestaje

It's likely, rather, to swoosh right by.

već se ubrzano nastavlja.

Now this has profound implications,

To ima duboke implikacije,

particularly when it comes to questions of power.

posebno kad se radi o pitanju moći.

For example, chimpanzees are strong --

Na primjer, čimpanze su snažne --

pound for pound, a chimpanzee is about twice as strong as a fit human male.

dvostruko su jače od muškaraca.

And yet, the fate of Kanzi and his pals depends a lot more

A ipak, sudbina Kanzija i njegovih prijatelja više ovisi o tome što

on what we humans do than on what the chimpanzees do themselves.

mi ljudi radimo nego o tome što same čimpanze rade.

Once there is superintelligence,

Jednom kada se stvori superinteligencija,

the fate of humanity may depend on what the superintelligence does.

sudbina će čovječanstva možda ovisiti o tome što ona radi.

Think about it:

Razmislite.

Machine intelligence is the last invention that humanity will ever need to make.

Strojna inteligencija je zadnji izum koji će čovječanstvo ikada trebati.

Machines will then be better at inventing than we are,

Strojevi će onda biti bolji u izumljivanju od nas,

and they'll be doing so on digital timescales.

i oni će to raditi u digitalnom vremenskom rasponu.

What this means is basically a telescoping of the future.

Načelno to znači teleskopiranje budućnosti.

Think of all the crazy technologies that you could have imagined

Zamislite sve moguće lude tehnologije

maybe humans could have developed in the fullness of time:

koje bi ljudi mogli razviti: lijek protiv starenja,

cures for aging, space colonization,

kolonizacija svemira,

self-replicating nanobots or uploading of minds into computers,

samoreplicirajući nanoboti, unošenje uma u računalo,

all kinds of science fiction-y stuff

svakakve znanstveno-fantastične stvari

that's nevertheless consistent with the laws of physics.

koje su u skladu sa zakonima fizike.

All of this superintelligence could develop, and possibly quite rapidly.

Superinteligencija mogla bi se razviti, i to dosta brzo.

Now, a superintelligence with such technological maturity

Superinteligencija s takvom tehnološkom zrelošću

would be extremely powerful,

bila bi iznimno moćna,

and at least in some scenarios, it would be able to get what it wants.

i bar bi u nekim slučajevima, mogla dobiti što želi.

We would then have a future that would be shaped by the preferences of this A.I.

Onda bismo u budućnosti imali svijet oblikovan prema preferencijama te UI.

Now a good question is, what are those preferences?

Dobro je pitanje koje su te preference.

Here it gets trickier.

Tu se stvari kompliciraju.

To make any headway with this,

Da bismo napredovali s tim,

we must first of all avoid anthropomorphizing.

moramo prvo izbjeći antropomorfizaciju.

And this is ironic because every newspaper article

A to je ironično jer svi novinski članci

about the future of A.I. has a picture of this:

o budućnosti UI imaju sliku ovoga:

So I think what we need to do is to conceive of the issue more abstractly,

Tako da mislim da trebamo pojmiti ovo pitanje malo apstraktnije,

not in terms of vivid Hollywood scenarios.

a ne u obliku holivudskih scenarija.

We need to think of intelligence as an optimization process,

Inteligenciju trebamo promatrati kao proces optimizacije,

a process that steers the future into a particular set of configurations.

proces koji usmjerava budućnost u određeni set konfiguracija.

A superintelligence is a really strong optimization process.

Superinteligencija je zaista jak proces optimizacije.

It's extremely good at using available means to achieve a state

Nevjerojatno je dobra u korištenju raspoloživih sredstava kako bi postigla

in which its goal is realized.

svoj cilj.

This means that there is no necessary connection between

To znači da nužno ne postoji povezanost između

being highly intelligent in this sense,

visoke inteligencije u tom smislu

and having an objective that we humans would find worthwhile or meaningful.

i cilja koji bismo mi ljudi smatrali vrijednim ili smislenim.

Suppose we give an A.I. the goal to make humans smile.

Recimo da UI damo cilj da nasmiju ljude.

When the A.I. is weak, it performs useful or amusing actions

Kad je UI slaba, izvodi korisne ili zabavne radnje

that cause its user to smile.

zbog kojih se njihov korisnik smije.

When the A.I. becomes superintelligent,

Kada UI postane superinteligentna,

it realizes that there is a more effective way to achieve this goal:

uvidjet će da postoji učinkovitiji način na koji postići taj cilj:

take control of the world

preuzimanje kontrole nad svijetom

and stick electrodes into the facial muscles of humans

i smještanje elektroda u mišiće lica ljudi

to cause constant, beaming grins.

kako bi izazvala neprestan, vedar osmijeh.

Another example,

Još jedan primjer,

suppose we give A.I. the goal to solve a difficult mathematical problem.

recimo da zadamo UI da riješi zahtjevan matematički problem.

When the A.I. becomes superintelligent,

Kad UI postane superinteligentna,

it realizes that the most effective way to get the solution to this problem

uviđa da je najučinkovitiji način za rješavanje toga problema

is by transforming the planet into a giant computer,

preobrazba planeta u ogromno računalo,

so as to increase its thinking capacity.

da bi povećala njegovu sposobnost razmišljanja.

And notice that this gives the A.I.s an instrumental reason

I imajte na umu da to daje UI instrumentalni razlog da

to do things to us that we might not approve of.

nam radi stvari koje mi možda ne odobravamo.

Human beings in this model are threats,

U ovom su primjeri ljudi prijetnja,

we could prevent the mathematical problem from being solved.

mogli bismo spriječiti rješavanje matematičkih problema.

Of course, perceivably things won't go wrong in these particular ways;

Naravno, stvari neće baš tako poći po krivu;

these are cartoon examples.

ovo su primjeri iz crtića.

But the general point here is important:

Ali važna je poanta:

if you create a really powerful optimization process

ako stvorite zaista moćan proces optimizacije

to maximize for objective x,

kako biste maksimizirali cilj x,

you better make sure that your definition of x

morate biti sigurni da vaša definicija x-a

incorporates everything you care about.

uključuje sve do čega vam je stalo.

This is a lesson that's also taught in many a myth.

Ovo je lekcija na koju nas upozoravaju mnogi mitovi.

King Midas wishes that everything he touches be turned into gold.

Kralj Mida želi da se sve što dotakne pretvori u zlato.

He touches his daughter, she turns into gold.

Dodirne kćer i ona se pretvori u zlato.

He touches his food, it turns into gold.

Dodirne hranu, ona postane zlato.

This could become practically relevant,

Ovo bi moglo postati i praktički važno,

not just as a metaphor for greed,

ne samo kao metafora za pohlepu,

but as an illustration of what happens

nego i kao prikaz onoga što se dogodi

if you create a powerful optimization process

ako stvorite moćan proces optimizacije

and give it misconceived or poorly specified goals.

i date mu pogrešne ili loše određene ciljeve.

Now you might say, if a computer starts sticking electrodes into people's faces,

Mogli biste reći da, ako računalo počne stavljati elektrode u lica ljudi,

we'd just shut it off.

jednostavno ćemo ga isključiti.

A, this is not necessarily so easy to do if we've grown dependent on the system --

Pod a, to nije tako jednostavno ako smo već ovisni o sustavu --

like, where is the off switch to the Internet?

na primjer, kako isključiti internet?

B, why haven't the chimpanzees flicked the off switch to humanity,

Pod b, zašto čimpanze nisu isključile prekidač ljudske vrste,

or the Neanderthals?

oni ili neandertalci?

They certainly had reasons.

Zasigurno su postojali razlozi za to.

We have an off switch, for example, right here.

Mi imamo prekidač, recimo, upravo ovdje.

(Choking)

(Guši se)

The reason is that we are an intelligent adversary;

Razlog je taj što smo mi inteligentni protivnici;

we can anticipate threats and plan around them.

predviđamo prijetnje i planiramo kako ih izbjeći.

But so could a superintelligent agent,

Ali to bi mogao raditi i superinteligentni izvršitelj

and it would be much better at that than we are.

i to puno bolje nego što mi to radimo.

The point is, we should not be confident that we have this under control here.

Poanta je ovoga da ne smijemo biti uvjereni da sve držimo pod kontrolom.

And we could try to make our job a little bit easier by, say,

Mogli bismo si olakšati posao tako što

putting the A.I. in a box,

bismo smjestili UI u kutiju,

like a secure software environment,

nekakvo sigurno softversko okruženje,

a virtual reality simulation from which it cannot escape.

virtualnu simulaciju stvarnosti iz koje ne može pobjeći.

But how confident can we be that the A.I. couldn't find a bug.

Ali kako možemo biti sigurni da UI neće pronaći rupu?

Given that merely human hackers find bugs all the time,

Ako uobzirimo da hakeri neprestano pronalaze rupe,

I'd say, probably not very confident.

vjerojatno ne možemo biti sigurni u to.

So we disconnect the ethernet cable to create an air gap,

Isključimo Ethernet kabel da bismo stvorili zračni prostor,

but again, like merely human hackers

ali opet, ljudski hakeri rutinski nadilaze

routinely transgress air gaps using social engineering.

te prostore socijalnim inženjeringom.

Right now, as I speak,

Dok ovo govorim,

I'm sure there is some employee out there somewhere

siguran sam da su negdje neku zaposlenicu

who has been talked into handing out her account details

nagovorili da otkrije detalje svog računa

by somebody claiming to be from the I.T. department.

nekome tko je tvrdio da je iz IT odjela.

More creative scenarios are also possible,

Mogući su i kreativniji scenariji,

like if you're the A.I.,

na primjer, ako ste UI,

you can imagine wiggling electrodes around in your internal circuitry

možete zamisliti micanje elektroda u unutrašnjem sklopu kako biste

to create radio waves that you can use to communicate.

stvorili radio valove kojima možete komunicirati.

Or maybe you could pretend to malfunction,

Ili se možete prevarati da ste neispravni

and then when the programmers open you up to see what went wrong with you,

i kada vas programeri otvore da vide što nije u redu s vama,

they look at the source code -- Bam! --

pogledaju izvorni kod i -- Bum! --

the manipulation can take place.

manipulacija je počela.

Or it could output the blueprint to a really nifty technology,

Ili bi moglo dati nacrt zaista divne tehnologije,

and when we implement it,

a kada ga primijenimo,

it has some surreptitious side effect that the A.I. had planned.

ima neke skrivene posljedice koje je UI isplanirala.

The point here is that we should not be confident in our ability

Poanta jest ta da ne bismo trebali biti presigurni u svoju sposobnost

to keep a superintelligent genie locked up in its bottle forever.

da superinteligentnog duha možemo zauvijek čuvati u boci.

Sooner or later, it will out.

Prije ili kasnije će se osloboditi.

I believe that the answer here is to figure out

Vjerujem da je rješenje u pronalasku načina kako

how to create superintelligent A.I. such that even if -- when -- it escapes,

stvoriti superinteligentnu UI koja bi čak - kad - se oslobodi

it is still safe because it is fundamentally on our side

još uvijek bila sigurna jer je u osnovi na našoj strani

because it shares our values.

jer dijeli naša uvjerenja.

I see no way around this difficult problem.

Ne vidim kako drukčije riješiti ovaj veliki problem.

Now, I'm actually fairly optimistic that this problem can be solved.

Zapravo sam ja i optimističan kada mislim da je problem rješiv.

We wouldn't have to write down a long list of everything we care about,

Ne bismo morali pisati duge liste svega do čega nam je stalo,

or worse yet, spell it out in some computer language

ili, još gore, unijeti to u neki računalni jezik

like C++ or Python,

kao C++ ili Python,

that would be a task beyond hopeless.

to bi bio i više nego beznadan pothvat!

Instead, we would create an A.I. that uses its intelligence

Umjesto toga, stvorimo UI koja koristi svoju inteligenciju kako

to learn what we value,

bi naučila što nam je važno,

and its motivation system is constructed in such a way that it is motivated

i čiji je motivacijski sustav konstruiran tako da bude motivirana za

to pursue our values or to perform actions that it predicts we would approve of.

provođenje naših uvjerenja ili izvođenje radnji za koje predviđa da ih odobravamo.

We would thus leverage its intelligence as much as possible

Tako bismo iskoristili njezinu inteligenciju što je više moguće

to solve the problem of value-loading.

kako bismo riješili ovaj problem.

This can happen,

To se može dogoditi,

and the outcome could be very good for humanity.

a ishod bi mogao biti jako dobar za čovječanstvo.

But it doesn't happen automatically.

Ali to se ne događa automatski.

The initial conditions for the intelligence explosion

Polazni uvjeti za eksploziju inteligencije

might need to be set up in just the right way

morat će biti postavljeni na točno određen način

if we are to have a controlled detonation.

ako želimo kontroliranu detonaciju.

The values that the A.I. has need to match ours,

Vrijednosti UI moraju odgovarati našima,

not just in the familiar context,

ne samo u poznatim kontekstima

like where we can easily check how the A.I. behaves,

u kojima lako provjerimo kako se UI ponaša,

but also in all novel contexts that the A.I. might encounter

nego i u novim kontekstima s kojima bi se UI mogla susresti

in the indefinite future.

u neodređenoj budućnosti.

And there are also some esoteric issues that would need to be solved, sorted out:

Postoje i neka ezoterična pitanja koja bi se morala riješiti:

the exact details of its decision theory,

točni detalji teorije odlučivanja,

how to deal with logical uncertainty and so forth.

kako se nositi s logičkom nesigurnosti i tako dalje.

So the technical problems that need to be solved to make this work

Tehnički problemi koji se moraju riješiti kako bi ovo uspjelo

look quite difficult --

poprilično su veliki --

not as difficult as making a superintelligent A.I.,

ne toliko veliki kao stvaranje superinteligentne UI,

but fairly difficult.

ali prilično veliki.

Here is the worry:

Ovo je razlog za brigu:

Making superintelligent A.I. is a really hard challenge.

Stvaranje superinteligentne UI težak je izazov.

Making superintelligent A.I. that is safe

Stvaranje sigurne superinteligentne UI

involves some additional challenge on top of that.

dodaje još izazova.

The risk is that if somebody figures out how to crack the first challenge

Rizik leži u otkrivanju kako savladati prvi izazov,

without also having cracked the additional challenge

ali bez shvaćanja kako savladati dodatni izazov,

of ensuring perfect safety.

osiguravanje sigurnosti.

So I think that we should work out a solution

Smatram da bismo trebali riješiti

to the control problem in advance,

pitanje kontrole unaprijed

so that we have it available by the time it is needed.

kako bismo bili spremni kada za to dođe vrijeme.

Now it might be that we cannot solve the entire control problem in advance

Može se dogoditi da ne možemo taj problem riješiti unaprijed

because maybe some elements can only be put in place

jer se mogu posložiti samo neki elementi

once you know the details of the architecture where it will be implemented.

kada točno saznamo gdje će se provoditi.

But the more of the control problem that we solve in advance,

Što više problema po pitanju kontrole riješimo unaprijed,

the better the odds that the transition to the machine intelligence era

bolji su izgledi da će prijelaz u eru strojne inteligencije

will go well.

proći dobro.

This to me looks like a thing that is well worth doing

Meni se čini da se to isplati raditi

and I can imagine that if things turn out okay,

i mogu zamisliti da, ako sve prođe u redu,

that people a million years from now look back at this century

će ljudi za milijun godina gledati na ovo stoljeće

and it might well be that they say that the one thing we did that really mattered

i reći da smo napravili jednu stvar koja je stvarno važna, a ta je

was to get this thing right.

da smo ovo napravili kako treba.

Thank you.

Hvala.

(Applause)

(Pljesak)