What happens when our computers get smarter than we are? | Nick Bostrom

翻译人员: Huazhe Xie 校对人员: Geoff Chen

I work with a bunch of mathematicians, philosophers and computer scientists,

我和一些数学家、 哲学家和电脑学家一起工作，

and we sit around and think about the future of machine intelligence,

我们会坐在一起思考未来的机械智能，

among other things.

和其他的一些事情。

Some people think that some of these things are sort of science fiction-y,

有的人认为这类事情只是科幻，

far out there, crazy.

不切实际，很疯狂。

But I like to say,

但是我想说，

okay, let's look at the modern human condition.

好吧，那我们来看看人类现状吧。

(Laughter)

（笑）

This is the normal way for things to be.

这是世间一种常态。

But if we think about it,

但是如果我们去思考，

we are actually recently arrived guests on this planet,

我们人类，其实相当晚才

the human species.

出现在这个星球上。

Think about if Earth was created one year ago,

想一想，如果地球是一年前才被创造的，

the human species, then, would be 10 minutes old.

人类，那么，10分钟前才出现。

The industrial era started two seconds ago.

然后工业时代两秒钟前刚刚开始。

Another way to look at this is to think of world GDP over the last 10,000 years,

另一种看待这件事的方式是去 想一下在过去一万年间的世界 GDP 状况。

I've actually taken the trouble to plot this for you in a graph.

我其实真的试着去做了一个统计图。

It looks like this.

就是这样。

(Laughter)

（笑）

It's a curious shape for a normal condition.

这是个令人好奇的形状，正常情况下。

I sure wouldn't want to sit on it.

我确定我不想坐在上面。

(Laughter)

（笑）

Let's ask ourselves, what is the cause of this current anomaly?

让我们扪心自问，到底是什么造成了 如此不寻常的现状？

Some people would say it's technology.

一些人会说，因为科技。

Now it's true, technology has accumulated through human history,

对于现在来说是对的，科技是人类历史 不断积累下来的果实。

and right now, technology advances extremely rapidly --

现在，科技发展十分迅速：

that is the proximate cause,

这是个直接原因，

that's why we are currently so very productive.

这就是为什么我们现在生产效率如此高。

But I like to think back further to the ultimate cause.

但是我想探究更远的在未来的终极原因。

Look at these two highly distinguished gentlemen:

看这两个非常不同的男士：

We have Kanzi --

这是 Kanzi，

he's mastered 200 lexical tokens, an incredible feat.

他已经掌握了 200 个词法标记， 一个难以置信的成就。

And Ed Witten unleashed the second superstring revolution.

Ed Witten 开创了第二个令人惊人的创新。

If we look under the hood, this is what we find:

如果我们去看这些事物的本质， 这是我们的发现：

basically the same thing.

全都是一样的。

One is a little larger,

一个稍微大了一点，

it maybe also has a few tricks in the exact way it's wired.

也许它有一些特殊的技巧。

These invisible differences cannot be too complicated, however,

但是，这些隐形的不同并没有很错综复杂，

because there have only been 250,000 generations

因为在我们和我们的祖先之间

since our last common ancestor.

只有 25 万代人。

We know that complicated mechanisms take a long time to evolve.

我们知道复杂的机制 需要很长的时间来进化得到。

So a bunch of relatively minor changes

所以，一些相对小的变化，

take us from Kanzi to Witten,

让我们从 Kanzi 变成了 Witten，

from broken-off tree branches to intercontinental ballistic missiles.

从捡起掉下的树枝作为武器， 到发射洲际导弹。

So this then seems pretty obvious that everything we've achieved,

因此，至今我们所办到的所有事情，

and everything we care about,

以及我们所关心的事情，

depends crucially on some relatively minor changes that made the human mind.

都取决于人大脑中细小的变化。

And the corollary, of course, is that any further changes

因此得出的结论是：在未来，

that could significantly change the substrate of thinking

任何显著的思考基体的变化，

could have potentially enormous consequences.

都能带来巨大的后果。

Some of my colleagues think we're on the verge

我的一些同事觉得我们即将会发明，

of something that could cause a profound change in that substrate,

足以深深地改变人类思考模式的科技。

and that is machine superintelligence.

就是超级机能智慧。

Artificial intelligence used to be about putting commands in a box.

以前的人工智慧 是把指令输入到一个箱子里。

You would have human programmers

你需要人类程序员，

that would painstakingly handcraft knowledge items.

来努力把知识转变成程序。

You build up these expert systems,

你会建立起一些专业系统，

and they were kind of useful for some purposes,

它们有时候会有帮助，

but they were very brittle, you couldn't scale them.

但是它们很生硬，你不能延展它们的功能。

Basically, you got out only what you put in.

基本上你只能得到你放进去的东西。

But since then,

但是自从那时候开始，

a paradigm shift has taken place in the field of artificial intelligence.

人工智能的领域发生了巨大的改变。

Today, the action is really around machine learning.

现在主要的研究方向是机器的学习。

So rather than handcrafting knowledge representations and features,

所以，预期设计出知识的再现，

we create algorithms that learn, often from raw perceptual data.

我们写出具有从原始感官数据学习的程序，

Basically the same thing that the human infant does.

像婴儿一样。

The result is A.I. that is not limited to one domain --

结果就不会局限于某个领域的人工智能：

the same system can learn to translate between any pairs of languages,

同一个系统可以学习两种语言之间的翻译

or learn to play any computer game on the Atari console.

或者学着玩 Atari 的游戏。

Now of course,

当然，现在，

A.I. is still nowhere near having the same powerful, cross-domain

人工智能还未能达到向人类一样，

ability to learn and plan as a human being has.

具有强大的跨领域学习能力。

The cortex still has some algorithmic tricks

人类大脑还具有一些运算技巧，

that we don't yet know how to match in machines.

可是我们不知道如何 将这些技巧用于机器。

So the question is,

所以我们现在需要问：

how far are we from being able to match those tricks?

我们还要多久才可以 让机器复制这种能力？

A couple of years ago,

几年前，

we did a survey of some of the world's leading A.I. experts,

我们对世界顶尖的人工智能专家 做了一次问卷调查，

to see what they think, and one of the questions we asked was,

来收集他们的想法，其中一道题目是：

"By which year do you think there is a 50 percent probability

“到哪一年你觉得人类会有 50% 的可能性

that we will have achieved human-level machine intelligence?"

创造达到人类水平的人工智能？”

We defined human-level here as the ability to perform

我们把这样的人工智能定义为

almost any job at least as well as an adult human,

有能力将任何任务 完成得至少和一名成年人一样好。

so real human-level, not just within some limited domain.

所以是真正的人类级别， 而不是仅限于一些领域。

And the median answer was 2040 or 2050,

而答案的中位数是 2040 到 2050 年，

depending on precisely which group of experts we asked.

取决于这些专家的群体。

Now, it could happen much, much later, or sooner,

当然这个有可能要过很久才能实现， 也有可能提前实现。

the truth is nobody really knows.

没有人知道确切的时间。

What we do know is that the ultimate limit to information processing

我们知道的事， 处理信息的能力的最终点，

in a machine substrate lies far outside the limits in biological tissue.

比任何生物组织要大很多。

This comes down to physics.

这取决与物理原理。

A biological neuron fires, maybe, at 200 hertz, 200 times a second.

一个生物神经元所发出的脉冲频率 大约位于 200 赫兹，每秒 200 次。

But even a present-day transistor operates at the Gigahertz.

但是就算是现在的电晶体 都以千兆赫的频率运行。

Neurons propagate slowly in axons, 100 meters per second, tops.

神经元在轴突中传输的速度较慢， 最多 100 米每秒。

But in computers, signals can travel at the speed of light.

但在电脑里，信号是以光速传播的。

There are also size limitations,

另外还有尺寸的限制，

like a human brain has to fit inside a cranium,

就像人类的大脑只能有颅骨那么大，

but a computer can be the size of a warehouse or larger.

但是一个电脑可以和仓库一样大，甚至更大。

So the potential for superintelligence lies dormant in matter,

因此超级智慧的潜能正潜伏在物质之中，

much like the power of the atom lay dormant throughout human history,

就像原子能潜伏在人类历史中一样，

patiently waiting there until 1945.

直到 1945。

In this century,

在这个世纪里，

scientists may learn to awaken the power of artificial intelligence.

科学家可能能将人工智慧的力量唤醒。

And I think we might then see an intelligence explosion.

那时候我觉得我们会看到智慧大爆发。

Now most people, when they think about what is smart and what is dumb,

大部分的人，当他们想 什么是聪明什么是笨的时候，

I think have in mind a picture roughly like this.

他们脑子里的画面是这样的：

So at one end we have the village idiot,

一边是村子里的傻子，

and then far over at the other side

一边是

we have Ed Witten, or Albert Einstein, or whoever your favorite guru is.

Ed Witten 或 Albert Einstein， 或者其他大师。

But I think that from the point of view of artificial intelligence,

但是我觉得从人工智能的观点来看，

the true picture is actually probably more like this:

真正的画面也许是这样：

AI starts out at this point here, at zero intelligence,

人工智能从这一点开始，零智慧。

and then, after many, many years of really hard work,

然后，在许多许多辛劳工作后，

maybe eventually we get to mouse-level artificial intelligence,

也许最终我们能达到老鼠级别的智慧，

something that can navigate cluttered environments

能在混乱中找到开出一条道路，

as well as a mouse can.

像一只老鼠一样。

And then, after many, many more years of really hard work, lots of investment,

之后，在更多更多年的辛苦研究 和投资之后，

maybe eventually we get to chimpanzee-level artificial intelligence.

也许最终我们能到达黑猩猩级人工智能。

And then, after even more years of really, really hard work,

在后来，更多年的研究之后，

we get to village idiot artificial intelligence.

我们能够达到村里的傻子级别的人工智能。

And a few moments later, we are beyond Ed Witten.

在一段时间之后， 我们能超越 Ed Witten。

The train doesn't stop at Humanville Station.

这列火车不会在“人类站”就停下。

It's likely, rather, to swoosh right by.

它比较可能会呼啸而过。

Now this has profound implications,

现在这个有深远的寓意，

particularly when it comes to questions of power.

尤其是当我们谈到力量权利的时候。

For example, chimpanzees are strong --

比如，黑猩猩很强壮：

pound for pound, a chimpanzee is about twice as strong as a fit human male.

同等的体重，一个黑猩猩是 两个健康男性那么强壮。

And yet, the fate of Kanzi and his pals depends a lot more

然而，Kanzi 和他的朋友们的命运 更多取决于

on what we humans do than on what the chimpanzees do themselves.

我们人类能做到什么， 而不是猩猩能做到什么。

Once there is superintelligence,

当超级智慧出现的时候，

the fate of humanity may depend on what the superintelligence does.

人类的命运也许会取决于 那个超级智慧体要做什么。

Think about it:

想一想：

Machine intelligence is the last invention that humanity will ever need to make.

机器智慧是人类需要创造的最后一个东西。

Machines will then be better at inventing than we are,

机器在那之后会比我们更擅长创造，

and they'll be doing so on digital timescales.

他们也会在数位时间里这样做。

What this means is basically a telescoping of the future.

这个意味着一个被缩短的未来。

Think of all the crazy technologies that you could have imagined

想一下你曾想象过的所有的疯狂的科技，

maybe humans could have developed in the fullness of time:

也许人类可以在适当的时候完成：

cures for aging, space colonization,

终结衰老、宇宙殖民、

self-replicating nanobots or uploading of minds into computers,

自我复制的纳米机器人 和大脑到电脑的传输，

all kinds of science fiction-y stuff

诸如此类的看似仅存在于科幻

that's nevertheless consistent with the laws of physics.

却有同时符合物理法则的元素。

All of this superintelligence could develop, and possibly quite rapidly.

超级智慧有办法开发出这些东西，也许更快。

Now, a superintelligence with such technological maturity

现在，一个拥有如此成熟科技的超级智慧体

would be extremely powerful,

将会是非常强大，

and at least in some scenarios, it would be able to get what it wants.

至少在一些情况下， 它能得到它想要的东西。

We would then have a future that would be shaped by the preferences of this A.I.

我们的未来就将会被 这个超级智慧体的喜好所主宰。

Now a good question is, what are those preferences?

现在的问题就是， 这些喜好是什么呢？

Here it gets trickier.

这很棘手。

To make any headway with this,

要在这个领域取得进步，

we must first of all avoid anthropomorphizing.

我们必须避免将机器智慧人格化。

And this is ironic because every newspaper article

这一点很讽刺， 因为每一个关于人工智能的未来

about the future of A.I. has a picture of this:

的新闻报道，都会有这个图片：

So I think what we need to do is to conceive of the issue more abstractly,

所以我觉得我们必须 用更抽象的方法看待这个问题，

not in terms of vivid Hollywood scenarios.

而不是在好莱坞电影的叙事之下。

We need to think of intelligence as an optimization process,

我们需要把智慧看做是一个优化的过程，

a process that steers the future into a particular set of configurations.

一个能把未来引导至 一个特殊组合结构的过程。

A superintelligence is a really strong optimization process.

一个超级智慧体是一个 非常强大的优化过程。

It's extremely good at using available means to achieve a state

它将会擅长利用资源来

达到自己的目标。

in which its goal is realized.

This means that there is no necessary connection between

这意味着有着高智慧和

being highly intelligent in this sense,

拥有一个对人类来说有用的目标之间

and having an objective that we humans would find worthwhile or meaningful.

并没有必然的联系。

Suppose we give an A.I. the goal to make humans smile.

假设我们给予人工智慧的目的是让人笑，

When the A.I. is weak, it performs useful or amusing actions

当人工智能弱的时候， 它能做出有用或好笑的表演，

that cause its user to smile.

这样它的使用者就会笑了。

When the A.I. becomes superintelligent,

当人工智能变成超级智慧体的时候，

it realizes that there is a more effective way to achieve this goal:

它会意识到有一个更有效的办法 能达到这个效果：

take control of the world

控制世界，

and stick electrodes into the facial muscles of humans

在人类面部肌肉上插入电极

to cause constant, beaming grins.

来让人类不断地笑。

Another example,

另一个例子：

suppose we give A.I. the goal to solve a difficult mathematical problem.

假设我们给予人工智能的目标 是解出很难的数学题，

When the A.I. becomes superintelligent,

当人工智能变成超级智慧体的时候，

it realizes that the most effective way to get the solution to this problem

它意识到有一个更有效的办法来解出问题，

is by transforming the planet into a giant computer,

是把整个地球变成一个巨型电脑，

so as to increase its thinking capacity.

这样它的运算能力就变更强大了。

And notice that this gives the A.I.s an instrumental reason

注意到这个是 给予人工智能一个模式型的理由

to do things to us that we might not approve of.

来做我们也许并不认可的事情。

Human beings in this model are threats,

人类在这个模式中是威胁，

we could prevent the mathematical problem from being solved.

我们可以人为地 让这个数学问题不能被解出。

Of course, perceivably things won't go wrong in these particular ways;

当然了，我们预见 这种事情不会错到这样的地步，

these are cartoon examples.

这些是夸张的例子。

But the general point here is important:

但是它们所代表的主旨很重要：

if you create a really powerful optimization process

如果你创造了一个非常强大的优化过程

to maximize for objective x,

来最大化目标 X，

you better make sure that your definition of x

你最好保证你的意义上的 X

incorporates everything you care about.

包括了所有你在乎的事情。

This is a lesson that's also taught in many a myth.

这是一个很多神话故事中都在传递的寓意。

King Midas wishes that everything he touches be turned into gold.

（希腊神话中）的 Midas 国王 希望他碰到的所有东西都能变成金子。

He touches his daughter, she turns into gold.

他碰到了他的女儿，她于是变成了金子。

He touches his food, it turns into gold.

他碰到了食物，于是食物变成了金子。

This could become practically relevant,

这个故事和我们的话题息息相关，

not just as a metaphor for greed,

并不只是因为它隐藏在对贪婪的暗喻，

but as an illustration of what happens

也是因为他指出了

if you create a powerful optimization process

如果你创造出来一个强大的优化过程

and give it misconceived or poorly specified goals.

并且给他了一个错误的或者不精确的目标， 后果会是什么。

Now you might say, if a computer starts sticking electrodes into people's faces,

现在也许你会说， 如果一个电脑开始在人类脸上插电极，

we'd just shut it off.

我们会关掉它。

A, this is not necessarily so easy to do if we've grown dependent on the system --

第一，这不是一件容易事， 如果我们变得非常依赖这个系统：

like, where is the off switch to the Internet?

比如，你知道互联网的开关在哪吗？

B, why haven't the chimpanzees flicked the off switch to humanity,

第二，为什么当初黑猩猩 没有关掉人类的开关，

or the Neanderthals?

或者尼安德特人的开关？

They certainly had reasons.

他们肯定有理由。

We have an off switch, for example, right here.

我们有一个开关，比如，这里。

(Choking)

（窒息声）

The reason is that we are an intelligent adversary;

之所以我们是聪明的敌人，

we can anticipate threats and plan around them.

因为我们可以预见到威胁并且尝试避免它。

But so could a superintelligent agent,

但是一个超级智慧体也可以，

and it would be much better at that than we are.

而且会做得更好。

The point is, we should not be confident that we have this under control here.

我们不应该很自信地 表示我们能控制所有事情。

And we could try to make our job a little bit easier by, say,

为了把我们的工作变得更简单一点， 我们应该试着，比如，

putting the A.I. in a box,

把人工智能放进一个小盒子，

like a secure software environment,

想一个保险的软件环境，

a virtual reality simulation from which it cannot escape.

一个它无法逃脱的虚拟现实模拟器。

But how confident can we be that the A.I. couldn't find a bug.

但是我们有信心它不可能能发现一个漏洞， 能让它逃出的漏洞吗？

Given that merely human hackers find bugs all the time,

连人类黑客每时每刻都能发现网络漏洞，

I'd say, probably not very confident.

我会说，也许不是很有信心。

So we disconnect the ethernet cable to create an air gap,

所以我们断开以太网的链接来创建一个空隙，

but again, like merely human hackers

但是，重申一遍，人类黑客都可以

routinely transgress air gaps using social engineering.

一次又一次以社会工程跨越这样的空隙。

Right now, as I speak,

现在，在我说话的时候，

I'm sure there is some employee out there somewhere

我肯定在这边的某个雇员，

who has been talked into handing out her account details

曾近被要求交出他的账户明细，

by somebody claiming to be from the I.T. department.

给一个自称是电脑信息部门的人。

More creative scenarios are also possible,

其他的情况也有可能，

like if you're the A.I.,

比如如果你是人工智能，

you can imagine wiggling electrodes around in your internal circuitry

你可以想象你用在你的体内 环绕复杂缠绕的电极

to create radio waves that you can use to communicate.

创造出一种无线电波来交流。

Or maybe you could pretend to malfunction,

或者也许你可以假装你出了问题，

and then when the programmers open you up to see what went wrong with you,

然后程序师就把你打开看看哪里出错了，

they look at the source code -- Bam! --

他们找出了源代码——Bang！——

the manipulation can take place.

你就可以取得控制权了。

Or it could output the blueprint to a really nifty technology,

或者它可以做出一个 非常漂亮的科技蓝图，

and when we implement it,

当我们实现之后，

it has some surreptitious side effect that the A.I. had planned.

它有一些被人工智能计划好的 秘密的副作用。

The point here is that we should not be confident in our ability

所以我们不能 对我们能够永远控制

to keep a superintelligent genie locked up in its bottle forever.

一个超级智能体的能力 表示过度自信。

Sooner or later, it will out.

在不久后，它会逃脱出来。

I believe that the answer here is to figure out

我相信我们需要弄明白

how to create superintelligent A.I. such that even if -- when -- it escapes,

如何创造出超级人工智能体，哪怕它逃走了，

it is still safe because it is fundamentally on our side

它仍然是无害的，因为它是我们这一边的，

because it shares our values.

因为它有我们的价值观。

I see no way around this difficult problem.

我认为这是个不可避免的问题。

Now, I'm actually fairly optimistic that this problem can be solved.

现在，我对这个问题能否被解决保持乐观。

We wouldn't have to write down a long list of everything we care about,

我们不需要写下 所有我们在乎的事情，

or worse yet, spell it out in some computer language

或者，更糟地，把这些事情变成计算机语言，

like C++ or Python,

C++ 或者 Python，

that would be a task beyond hopeless.

这是个不可能的任务。

Instead, we would create an A.I. that uses its intelligence

而是，我们会创造出一个人工智能机器人， 用它自己的智慧

to learn what we value,

来学习我们的价值观，

and its motivation system is constructed in such a way that it is motivated

它的激励制度可以激励它

to pursue our values or to perform actions that it predicts we would approve of.

来追求我们的价值观 或者去做我们会赞成的事情。

We would thus leverage its intelligence as much as possible

我们会因此最大地提高它的智力，

to solve the problem of value-loading.

来解决富有价值的问题。

This can happen,

这是有可能的，

and the outcome could be very good for humanity.

结果可以使人类非常受益。

But it doesn't happen automatically.

但它不是自动发生的。

The initial conditions for the intelligence explosion

智慧大爆炸的初始条件

might need to be set up in just the right way

需要被正确地建立起来，

if we are to have a controlled detonation.

如果我们想要一切在掌握之中。

The values that the A.I. has need to match ours,

人工智能的价值观 要和我们的价值观相辅相成，

not just in the familiar context,

不只是在熟悉的情况下，

like where we can easily check how the A.I. behaves,

比如当我们能很容易检查它的行为的时候，

but also in all novel contexts that the A.I. might encounter

但也要在所有人工智能可能会遇到的 前所未有的情况下，

in the indefinite future.

在没有界限的未来， 与我们的价值观相辅相成。

And there are also some esoteric issues that would need to be solved, sorted out:

也有很多深奥的问题需要被分拣解决：

the exact details of its decision theory,

它如何做决定，

how to deal with logical uncertainty and so forth.

如何解决逻辑不确定性和类似的情况。

So the technical problems that need to be solved to make this work

所以技术上的待解决问题让这个任务

look quite difficult --

看起来有些困难：

not as difficult as making a superintelligent A.I.,

并没有像做出一个超级智慧体一样困难，

but fairly difficult.

但是还是很难。

Here is the worry:

这使我们所担心的：

Making superintelligent A.I. is a really hard challenge.

创造出一个超级智慧体确实是个很大的挑战。

Making superintelligent A.I. that is safe

创造出一个安全的超级智慧体，

involves some additional challenge on top of that.

是个更大的挑战。

The risk is that if somebody figures out how to crack the first challenge

风险是，如果有人有办法解决第一个难题，

without also having cracked the additional challenge

却无法解决第二个

of ensuring perfect safety.

确保安全性的挑战。

So I think that we should work out a solution

所以我认为我们应该预先想出

to the control problem in advance,

“控制性”的解决方法，

so that we have it available by the time it is needed.

这样我们就能在需要的时候用到它了。

Now it might be that we cannot solve the entire control problem in advance

现在也许我们并不能 预先解决全部的控制性问题，

because maybe some elements can only be put in place

因为有些因素需要你了解

once you know the details of the architecture where it will be implemented.

你要应用到的那个构架的细节才能实施。

But the more of the control problem that we solve in advance,

但如果我们能解决更多控制性的难题，

the better the odds that the transition to the machine intelligence era

当我们迈入机器智能时代后

will go well.

就能更加顺利。

This to me looks like a thing that is well worth doing

这对于我来说是个值得一试的东西，

and I can imagine that if things turn out okay,

而且我能想象，如果一切顺利，

that people a million years from now look back at this century

几百万年后的人类回首我们这个世纪，

and it might well be that they say that the one thing we did that really mattered

他们也许会说， 我们所做的最最重要的事情，

was to get this thing right.

就是做了这个正确的决定。

Thank you.

谢谢。

(Applause)

（掌声）