3 principles for creating safer AI | Stuart Russell

翻译人员: Yichen Zheng 校对人员: Yanyan Hong

This is Lee Sedol.

这是李世石。

Lee Sedol is one of the world's greatest Go players,

李世石是全世界 最顶尖的围棋高手之一，

and he's having what my friends in Silicon Valley call

在这一刻，他所经历的 足以让我硅谷的朋友们

a "Holy Cow" moment --

喊一句”我的天啊“——

(Laughter)

（笑声）

a moment where we realize

在这一刻，我们意识到

that AI is actually progressing a lot faster than we expected.

原来人工智能发展的进程 比我们预想的要快得多。

So humans have lost on the Go board. What about the real world?

人们在围棋棋盘上已经输了， 那在现实世界中又如何呢？

当然了，现实世界要 比围棋棋盘要大得多，

Well, the real world is much bigger,

much more complicated than the Go board.

复杂得多。

It's a lot less visible,

相比之下每一步也没那么明确，

but it's still a decision problem.

但现实世界仍然是一个选择性问题。

And if we think about some of the technologies

如果我们想想那一些在不久的未来，

that are coming down the pike ...

即将来临的新科技……

Noriko [Arai] mentioned that reading is not yet happening in machines,

Noriko提到机器还不能进行阅读，

at least with understanding.

至少达不到理解的程度，

But that will happen,

但这迟早会发生，

而当它发生时，

and when that happens,

very soon afterwards,

不久之后，

机器就将读遍人类写下的所有东西。

machines will have read everything that the human race has ever written.

And that will enable machines,

这将使机器除了拥有

along with the ability to look further ahead than humans can,

比人类看得更远的能力，

as we've already seen in Go,

就像我们在围棋中看到的那样，

if they also have access to more information,

如果机器能接触到比人类更多的信息，

they'll be able to make better decisions in the real world than we can.

则将能够在现实世界中 做出比人类更好的选择。

So is that a good thing?

那这是一件好事吗？

Well, I hope so.

我当然希望如此。

Our entire civilization, everything that we value,

人类的全部文明， 我们所珍视的一切，

is based on our intelligence.

都是基于我们的智慧之上。

And if we had access to a lot more intelligence,

如果我们能掌控更强大的智能，

then there's really no limit to what the human race can do.

那我们人类的 创造力 就真的没有极限了。

And I think this could be, as some people have described it,

我认为这可能就像很多人描述的那样

the biggest event in human history.

会成为人类历史上最重要的事件。

So why are people saying things like this,

那为什么有的人会说出以下的言论，

that AI might spell the end of the human race?

说人工智能将是人类的末日呢？

Is this a new thing?

这是一个新事物吗？

Is it just Elon Musk and Bill Gates and Stephen Hawking?

这只关乎伊隆马斯克、 比尔盖茨，和斯提芬霍金吗？

Actually, no. This idea has been around for a while.

其实不是的，人工智能 这个概念已经存在很长时间了。

Here's a quotation:

请看这段话：

"Even if we could keep the machines in a subservient position,

“即便我们能够将机器 维持在一个屈服于我们的地位，

for instance, by turning off the power at strategic moments" --

比如说，在战略性时刻将电源关闭。”——

and I'll come back to that "turning off the power" idea later on --

我等会儿再来讨论 ”关闭电源“这一话题，

"we should, as a species, feel greatly humbled."

”我们，作为一个物种， 仍然应该自感惭愧。“

So who said this? This is Alan Turing in 1951.

这段话是谁说的呢？ 是阿兰图灵，他在1951年说的。

Alan Turing, as you know, is the father of computer science

阿兰图灵，众所皆知， 是计算机科学之父。

and in many ways, the father of AI as well.

从很多意义上说， 他也是人工智能之父。

So if we think about this problem,

当我们考虑这个问题，

the problem of creating something more intelligent than your own species,

创造一个比自己更智能的 物种的问题时，

we might call this "the gorilla problem,"

我们不妨将它称为”大猩猩问题“，

because gorillas' ancestors did this a few million years ago,

因为这正是大猩猩的 祖先们几百万年前所经历的。

and now we can ask the gorillas:

我们今天可以去问大猩猩们：

Was this a good idea?

那么做是不是一个好主意？

So here they are having a meeting to discuss whether it was a good idea,

在这幅图里，大猩猩们正在 开会讨论那么做是不是一个好主意，

and after a little while, they conclude, no,

片刻后他们下定结论，不是的。

this was a terrible idea.

那是一个很糟糕的主意。

我们的物种已经奄奄一息了，

Our species is in dire straits.

In fact, you can see the existential sadness in their eyes.

你都可以从它们的眼神中看到这种忧伤，

(Laughter)

（笑声）

So this queasy feeling that making something smarter than your own species

所以创造比你自己更聪明的物种，

is maybe not a good idea --

也许不是一个好主意——

what can we do about that?

那我们能做些什么呢？

Well, really nothing, except stop doing AI,

其实没什么能做的， 除了停止研究人工智能，

and because of all the benefits that I mentioned

但因为人工智能能带来 我之前所说的诸多益处，

and because I'm an AI researcher,

也因为我是 人工智能的研究者之一，

I'm not having that.

我可不同意就这么止步。

I actually want to be able to keep doing AI.

实际上，我想继续做人工智能。

So we actually need to nail down the problem a bit more.

所以我们需要把这个问题更细化一点，

它到底是什么呢？

What exactly is the problem?

Why is better AI possibly a catastrophe?

那就是为什么更强大的 人工智能可能会是灾难呢？

So here's another quotation:

再来看这段话：

"We had better be quite sure that the purpose put into the machine

”我们一定得确保我们 给机器输入的目的和价值

is the purpose which we really desire."

是我们确实想要的目的和价值。“

This was said by Norbert Wiener in 1960,

这是诺博特维纳在1960年说的，

shortly after he watched one of the very early learning systems

他说这话时是刚看到 一个早期的学习系统，

learn to play checkers better than its creator.

这个系统在学习如何能把 西洋棋下得比它的创造者更好。

But this could equally have been said

与此如出一辙的一句话，

by King Midas.

迈达斯国王也说过。

King Midas said, "I want everything I touch to turn to gold,"

迈达斯国王说：”我希望 我触碰的所有东西都变成金子。“

结果他真的获得了点石成金的能力。

and he got exactly what he asked for.

That was the purpose that he put into the machine,

那就是他所输入的目的，

so to speak,

从一定程度上说，

and then his food and his drink and his relatives turned to gold

后来他的食物、 他的家人都变成了金子，

and he died in misery and starvation.

他死在痛苦与饥饿之中。

So we'll call this "the King Midas problem"

我们可以把这个问题 叫做”迈达斯问题“，

of stating an objective which is not, in fact,

这个问题是我们阐述的目标，但实际上

truly aligned with what we want.

与我们真正想要的不一致，

In modern terms, we call this "the value alignment problem."

用现代的术语来说， 我们把它称为”价值一致性问题“。

Putting in the wrong objective is not the only part of the problem.

而输入错误的目标 仅仅是问题的一部分。

There's another part.

它还有另一部分。

If you put an objective into a machine,

如果你为机器输入一个目标，

even something as simple as, "Fetch the coffee,"

即便是一个很简单的目标， 比如说”去把咖啡端来“，

the machine says to itself,

机器会对自己说：

"Well, how might I fail to fetch the coffee?

”好吧，那我要怎么去拿咖啡呢？

Someone might switch me off.

说不定有人会把我的电源关掉。

OK, I have to take steps to prevent that.

好吧，那我要想办法 阻止别人把我关掉。

I will disable my 'off' switch.

我得让我的‘关闭’开关失效。

I will do anything to defend myself against interference

我得尽一切可能自我防御， 不让别人干涉我，

with this objective that I have been given."

这都是因为我被赋予的目标。”

So this single-minded pursuit

这种一根筋的思维，

in a very defensive mode of an objective that is, in fact,

以一种十分防御型的 模式去实现某一目标，

实际上与我们人类最初 想实现的目标并不一致——

not aligned with the true objectives of the human race --

that's the problem that we face.

这就是我们面临的问题。

And in fact, that's the high-value takeaway from this talk.

实际上，这就是今天这个演讲的核心。

If you want to remember one thing,

如果你在我的演讲中只记住一件事，

it's that you can't fetch the coffee if you're dead.

那就是：如果你死了， 你就不能去端咖啡了。

(Laughter)

（笑声）

It's very simple. Just remember that. Repeat it to yourself three times a day.

这很简单。记住它就行了。 每天对自己重复三遍。

(Laughter)

（笑声）

And in fact, this is exactly the plot

实际上，这正是电影

of "2001: [A Space Odyssey]"

《2001太空漫步》的剧情。

HAL has an objective, a mission,

HAL有一个目标，一个任务，

which is not aligned with the objectives of the humans,

但这个目标和人类的目标不一致，

and that leads to this conflict.

这就导致了矛盾的产生。

Now fortunately, HAL is not superintelligent.

幸运的是，HAL并不具备超级智能，

He's pretty smart, but eventually Dave outwits him

他挺聪明的，但还是 比不过人类主角戴夫，

and manages to switch him off.

戴夫成功地把HAL关掉了。

But we might not be so lucky.

但我们可能就没有这么幸运了。

So what are we going to do?

那我们应该怎么办呢？

I'm trying to redefine AI

我想要重新定义人工智能，

to get away from this classical notion

远离传统的定义，

of machines that intelligently pursue objectives.

将其仅限定为 机器通过智能去达成目标。

There are three principles involved.

新的定义涉及到三个原则：

第一个原则是利他主义原则，

The first one is a principle of altruism, if you like,

that the robot's only objective

也就是说，机器的唯一目标

is to maximize the realization of human objectives,

就是去最大化地实现人类的目标，

of human values.

人类的价值。

And by values here I don't mean touchy-feely, goody-goody values.

至于价值，我指的不是感情化的价值，

I just mean whatever it is that the human would prefer

而是指人类对生活所向往的，

their life to be like.

无论是什么。

And so this actually violates Asimov's law

这实际上违背了阿西莫夫定律，

that the robot has to protect its own existence.

他指出机器人一定要维护自己的生存。

It has no interest in preserving its existence whatsoever.

但我定义的机器 对维护自身生存毫无兴趣。

The second law is a law of humility, if you like.

第二个原则不妨称之为谦逊原则。

And this turns out to be really important to make robots safe.

这一条对于制造安全的机器十分重要。

It says that the robot does not know

它说的是机器不知道

what those human values are,

人类的价值是什么，

so it has to maximize them, but it doesn't know what they are.

机器知道它需要将人类的价值最大化， 却不知道这价值究竟是什么。

And that avoids this problem of single-minded pursuit

为了避免一根筋地追求

of an objective.

某一目标，

This uncertainty turns out to be crucial.

这种不确定性是至关重要的。

Now, in order to be useful to us,

那机器为了对我们有用，

it has to have some idea of what we want.

它就得掌握一些 关于我们想要什么的信息。

It obtains that information primarily by observation of human choices,

它主要通过观察人类 做的选择来获取这样的信息，

so our own choices reveal information

我们自己做出的选择会包含着

about what it is that we prefer our lives to be like.

关于我们希望我们的生活 是什么样的信息，

So those are the three principles.

这就是三条原则。

让我们来看看它们是如何应用到

Let's see how that applies to this question of:

"Can you switch the machine off?" as Turing suggested.

像图灵说的那样， “将机器关掉”这个问题上来。

So here's a PR2 robot.

这是一个PR2机器人。

我们实验室里有一个。

This is one that we have in our lab,

and it has a big red "off" switch right on the back.

它的背面有一个大大的红色的开关。

The question is: Is it going to let you switch it off?

那问题来了：它会让你把它关掉吗？

如果我们按传统的方法，

If we do it the classical way,

we give it the objective of, "Fetch the coffee, I must fetch the coffee,

给它一个目标，让它拿咖啡， 它会想：”我必须去拿咖啡，

I can't fetch the coffee if I'm dead,"

但我死了就不能拿咖啡了。“

so obviously the PR2 has been listening to my talk,

显然PR2听过我的演讲了，

and so it says, therefore, "I must disable my 'off' switch,

所以它说：”我必须让我的开关失灵，

and probably taser all the other people in Starbucks

可能还要把那些在星巴克里，

who might interfere with me."

可能干扰我的人都电击一下。“

(Laughter)

（笑声）

So this seems to be inevitable, right?

这看起来必然会发生，对吗？

This kind of failure mode seems to be inevitable,

这种失败看起来是必然的，

and it follows from having a concrete, definite objective.

因为机器人在遵循 一个十分确定的目标。

So what happens if the machine is uncertain about the objective?

那如果机器对目标 不那么确定会发生什么呢？

Well, it reasons in a different way.

那它的思路就不一样了。

It says, "OK, the human might switch me off,

它会说：”好的，人类可能会把我关掉，

but only if I'm doing something wrong.

但只在我做错事的时候。

Well, I don't really know what wrong is,

我不知道什么是错事，

但我知道我不该做那些事。”

but I know that I don't want to do it."

这就是第一和第二原则。

So that's the first and second principles right there.

"So I should let the human switch me off."

“那我就应该让人类把我关掉。”

And in fact you can calculate the incentive that the robot has

事实上你可以计算出机器人

to allow the human to switch it off,

让人类把它关掉的动机，

而且这个动机是

and it's directly tied to the degree

of uncertainty about the underlying objective.

与对目标的不确定程度直接相关的。

And then when the machine is switched off,

当机器被关闭后，

that third principle comes into play.

第三条原则就起作用了。

It learns something about the objectives it should be pursuing,

机器开始学习它所追求的目标，

because it learns that what it did wasn't right.

因为它知道它刚做的事是不对的。

In fact, we can, with suitable use of Greek symbols,

实际上，我们可以用希腊字母

as mathematicians usually do,

就像数学家们经常做的那样，

we can actually prove a theorem

直接证明这一定理，

that says that such a robot is provably beneficial to the human.

那就是这样的一个机器人 对人们是绝对有利的。

You are provably better off with a machine that's designed in this way

可以证明我们的生活 有如此设计的机器人会变得

than without it.

比没有这样的机器人更好。

So this is a very simple example, but this is the first step

这是一个很简单的例子，但这只是

in what we're trying to do with human-compatible AI.

我们尝试实现与人类 兼容的人工智能的第一步。

Now, this third principle,

现在来看第三个原则。

I think is the one that you're probably scratching your head over.

我知道你们可能正在 为这一个原则而大伤脑筋。

You're probably thinking, "Well, you know, I behave badly.

你可能会想：“你知道， 我有时不按规矩办事。

I don't want my robot to behave like me.

我可不希望我的机器人 像我一样行事。

我有时大半夜偷偷摸摸地 从冰箱里找东西吃，

I sneak down in the middle of the night and take stuff from the fridge.

I do this and that."

诸如此类的事。”

There's all kinds of things you don't want the robot doing.

有各种各样的事你是 不希望机器人去做的。

But in fact, it doesn't quite work that way.

但实际上并不一定会这样。

Just because you behave badly

仅仅是因为你表现不好，

doesn't mean the robot is going to copy your behavior.

并不代表机器人就会复制你的行为。

它会去尝试理解你做事的动机， 而且可能会在合适的情况下制止你去做

It's going to understand your motivations and maybe help you resist them,

if appropriate.

那些不该做的事。

But it's still difficult.

但这仍然十分困难。

What we're trying to do, in fact,

实际上，我们在做的是

is to allow machines to predict for any person and for any possible life

让机器去预测任何一个人， 在他们的任何一种

that they could live,

可能的生活中

and the lives of everybody else:

以及别人的生活中，

Which would they prefer?

他们会更倾向于哪一种？

And there are many, many difficulties involved in doing this;

这涉及到诸多困难；

I don't expect that this is going to get solved very quickly.

我不认为这会很快地就被解决。

The real difficulties, in fact, are us.

实际上，真正的困难是我们自己。

As I have already mentioned, we behave badly.

就像我刚说的那样， 我们做事不守规矩，

我们中有的人甚至行为肮脏。

In fact, some of us are downright nasty.

Now the robot, as I said, doesn't have to copy the behavior.

就像我说的， 机器人并不会复制那些行为，

The robot does not have any objective of its own.

机器人没有自己的目标，

It's purely altruistic.

它是完全无私的。

And it's not designed just to satisfy the desires of one person, the user,

它的设计不是去满足 某一个人、一个用户的欲望，

but in fact it has to respect the preferences of everybody.

而是去尊重所有人的意愿。

So it can deal with a certain amount of nastiness,

所以它能对付一定程度的肮脏行为。

and it can even understand that your nastiness, for example,

它甚至能理解你的不端行为，比如说

you may take bribes as a passport official

假如你是一个边境护照官员， 很可能收取贿赂，

because you need to feed your family and send your kids to school.

因为你得养家、 得供你的孩子们上学。

It can understand that; it doesn't mean it's going to steal.

机器人能理解这一点， 它不会因此去偷，

In fact, it'll just help you send your kids to school.

它反而会帮助你去供孩子们上学。

We are also computationally limited.

我们的计算能力也是有限的。

Lee Sedol is a brilliant Go player,

李世石是一个杰出的围棋大师，

but he still lost.

但他还是输了。

So if we look at his actions, he took an action that lost the game.

如果我们看他的行动， 他最终输掉了棋局。

That doesn't mean he wanted to lose.

但这不意味着他想要输。

So to understand his behavior,

所以要理解他的行为，

we actually have to invert through a model of human cognition

我们得从人类认知模型来反过来想，

that includes our computational limitations -- a very complicated model.

这包含了我们的计算能力限制， 是一个很复杂的模型，

But it's still something that we can work on understanding.

但仍然是我们可以尝试去理解的。

Probably the most difficult part, from my point of view as an AI researcher,

可能对于我这样一个 人工智能研究人员来说最大的困难，

是我们彼此各不相同。

is the fact that there are lots of us,

and so the machine has to somehow trade off, weigh up the preferences

所以机器必须想办法去判别衡量

of many different people,

不同人的不同需求，

and there are different ways to do that.

而又有众多方法去做这样的判断。

Economists, sociologists, moral philosophers have understood that,

经济学家、社会学家、 哲学家都理解这一点，

and we are actively looking for collaboration.

我们正在积极地去寻求合作。

Let's have a look and see what happens when you get that wrong.

让我们来看看如果我们 把这一步弄错了会怎么样。

So you can have a conversation, for example,

举例来说，你可能会 与你的人工智能助理，

with your intelligent personal assistant

有这样的对话：

that might be available in a few years' time.

这样的人工智能可能几年内就会出现，

Think of a Siri on steroids.

可以把它想做加强版的Siri。

So Siri says, "Your wife called to remind you about dinner tonight."

Siri对你说：“你的妻子打电话 提醒你今晚要跟她共进晚餐。”

And of course, you've forgotten. "What? What dinner?

而你呢，自然忘了这回事： “什么？什么晚饭？

What are you talking about?"

你在说什么？”

"Uh, your 20th anniversary at 7pm."

“啊，你们晚上7点， 庆祝结婚20周年纪念日。”

"I can't do that. I'm meeting with the secretary-general at 7:30.

“我可去不了。 我约了晚上7点半见领导。

How could this have happened?"

怎么会这样呢？”

"Well, I did warn you, but you overrode my recommendation."

“呃，我可是提醒过你的， 但你不听我的建议。”

"Well, what am I going to do? I can't just tell him I'm too busy."

“我该怎么办呢？我可不能 跟领导说我有事，没空见他。”

"Don't worry. I arranged for his plane to be delayed."

“别担心。我已经安排了， 让他的航班延误。

(Laughter)

（笑声）

"Some kind of computer malfunction."

“像是因为某种计算机故障那样。”

(Laughter)

（笑声）

"Really? You can do that?"

“真的吗？这个你也能做到？”

"He sends his profound apologies

“领导很不好意思，跟你道歉，

and looks forward to meeting you for lunch tomorrow."

并且告诉你明天 中午午饭不见不散。”

（笑声）

(Laughter)

这里就有一个小小的问题。

So the values here -- there's a slight mistake going on.

This is clearly following my wife's values

这显然是在遵循我妻子的价值论，

which is "Happy wife, happy life."

那就是“老婆开心，生活舒心”。

(Laughter)

（笑声）

It could go the other way.

它也有可能发展成另一种情况。

You could come home after a hard day's work,

你忙碌一天，回到家里，

and the computer says, "Long day?"

电脑对你说：“像是繁忙的一天啊？”

“是啊，我连午饭都没来得及吃。”

"Yes, I didn't even have time for lunch."

“那你一定很饿了吧。”

"You must be very hungry."

"Starving, yeah. Could you make some dinner?"

“快饿晕了。你能做点晚饭吗？”

"There's something I need to tell you."

“有一件事我得告诉你。

（笑声）

(Laughter)

"There are humans in South Sudan who are in more urgent need than you."

”南苏丹的人们可比你更需要照顾。

(Laughter)

（笑声）

“所以我要离开了。 你自己做饭去吧。”

"So I'm leaving. Make your own dinner."

(Laughter)

（笑声）

So we have to solve these problems,

我们得解决这些问题，

and I'm looking forward to working on them.

我也很期待去解决。

There are reasons for optimism.

我们有理由感到乐观。

One reason is,

理由之一是

there is a massive amount of data.

我们有大量的数据，

Because remember -- I said they're going to read everything

记住，我说过机器将能够阅读一切

人类所写下来的东西，

the human race has ever written.

而我们写下的大多数是 我们做的什么事情，

Most of what we write about is human beings doing things

以及其他人对此有什么意见。

and other people getting upset about it.

So there's a massive amount of data to learn from.

所以机器可以从大量的数据中去学习。

There's also a very strong economic incentive

同时从经济的角度， 我们也有足够的动机

to get this right.

去把这件事做对。

So imagine your domestic robot's at home.

想象一下，你家里有个居家机器人，

You're late from work again and the robot has to feed the kids,

而你又得加班， 机器人得给孩子们做饭,

and the kids are hungry and there's nothing in the fridge.

孩子们很饿， 但冰箱里什么都没有。

And the robot sees the cat.

然后机器人看到了家里的猫，

(Laughter)

（笑声）

And the robot hasn't quite learned the human value function properly,

机器人还没学透人类的价值论，

so it doesn't understand

所以它不知道

猫的感情价值 大于猫的营养价值。

the sentimental value of the cat outweighs the nutritional value of the cat.

（笑声）

(Laughter)

接下来会发生什么？

So then what happens?

Well, it happens like this:

差不多是这样的：

"Deranged robot cooks kitty for family dinner."

头版头条：“疯狂的机器人 把猫煮了给主人当晚饭！”

That one incident would be the end of the domestic robot industry.

这一个事故就足以结束 整个居家机器人产业。

So there's a huge incentive to get this right

所以我们有足够的动机在我们实现

long before we reach superintelligent machines.

超级智能机器让它更加完善。

So to summarize:

总结来说：

I'm actually trying to change the definition of AI

我想要改变人工智能的定义，

so that we have provably beneficial machines.

让我们可以证明机器对我们是有利的。

And the principles are:

这三个原则是：

machines that are altruistic,

机器是利他的，

that want to achieve only our objectives,

只想着实现我们的目标，

but that are uncertain about what those objectives are,

但它不确定我们的目标是什么，

所以它会观察我们，

and will watch all of us

to learn more about what it is that we really want.

从中学习我们想要的究竟是什么。

And hopefully in the process, we will learn to be better people.

希望在这个过程中， 我们也能学会成为更好的人。

Thank you very much.

谢谢大家。

(Applause)

（掌声）

Chris Anderson: So interesting, Stuart.

克里斯安德森： 非常有意思，斯图尔特。

我们趁着工作人员 为下一位演讲者布置的时候

We're going to stand here a bit because I think they're setting up

来简单聊几句。

for our next speaker.

A couple of questions.

我有几个问题。

So the idea of programming in ignorance seems intuitively really powerful.

从直觉上来看，将无知编入到程序中 似乎是一个很重要的理念，

当你要实现超级智能时，

As you get to superintelligence,

what's going to stop a robot

什么能阻止机器人？

reading literature and discovering this idea that knowledge

当它在阅读和学习的过程中发现，

is actually better than ignorance

知识比无知更强大，

and still just shifting its own goals and rewriting that programming?

然后就改变它的目标 去重新编写程序呢？

Stuart Russell: Yes, so we want it to learn more, as I said,

斯图尔特拉塞尔：是的， 我们想要它去学习，就像我说的，

about our objectives.

学习我们的目标。

It'll only become more certain as it becomes more correct,

它只有在理解得越来越正确的时候， 才会变得更确定，

so the evidence is there

我们有证据显示，

and it's going to be designed to interpret it correctly.

它的设计使它能按正确的方式理解。

It will understand, for example, that books are very biased

比如说，它能够理解书中的论证是

in the evidence they contain.

带有非常强的偏见的。

They only talk about kings and princes

书中只会讲述国王、王子

and elite white male people doing stuff.

和那些精英白人男性做的事。

So it's a complicated problem,

这是一个复杂的问题，

but as it learns more about our objectives

但当它更深入地学习我们的目标时，

it will become more and more useful to us.

它就变得对我们更有用。

CA: And you couldn't just boil it down to one law,

CA：那你不能把这些 都集中在一条准则里吗？

you know, hardwired in:

把这样的命令写在它的程序里：

"if any human ever tries to switch me off,

“如果人类什么时候想把我关掉，

I comply. I comply."

我服从。我服从。”

SR: Absolutely not.

SR：绝对不行，

那将是一个很糟糕的主意。

That would be a terrible idea.

So imagine that you have a self-driving car

试想一下，你有一辆无人驾驶汽车，

and you want to send your five-year-old

你想让它送你五岁的孩子

off to preschool.

去上学。

Do you want your five-year-old to be able to switch off the car

你希望你五岁的孩子 能在汽车运行过程中

将它关闭吗？

while it's driving along?

应该不会吧。

Probably not.

它得理解下指令的人有多理智， 是不是讲道理。

So it needs to understand how rational and sensible the person is.

The more rational the person,

这个人越理智，

the more willing you are to be switched off.

它就越愿意自己被关掉。

If the person is completely random or even malicious,

如果这个人是完全思绪混乱 或者甚至是有恶意的，

then you're less willing to be switched off.

那你就不愿意它被关掉。

CA: All right. Stuart, can I just say,

CA：好吧。斯图尔特，我得说

I really, really hope you figure this out for us.

我真的希望你为我们 能把这一切研究出来，

很感谢你的演讲，太精彩了。

Thank you so much for that talk. That was amazing.

SR: Thank you.

SR：谢谢。

(Applause)

（掌声）