3 principles for creating safer AI | Stuart Russell

譯者: 易帆 余 審譯者: Wilde Luo

This is Lee Sedol.

這是李世石。

Lee Sedol is one of the world's greatest Go players,

李世石是全世界 頂尖圍棋高手之一，

and he's having what my friends in Silicon Valley call

此時，他正在經歷的是 我的矽谷朋友們稱之為

a "Holy Cow" moment --

「我的媽呀！」的時刻......

(Laughter)

（笑聲）

a moment where we realize

在這一刻讓我們意識到，

that AI is actually progressing a lot faster than we expected.

原來人工智慧發展的進程 比我們預期的要快得多。

So humans have lost on the Go board. What about the real world?

人類已在圍棋博弈中落敗， 那現實世界中情況又如何？

Well, the real world is much bigger,

當然啦，現實世界要比棋盤 廣闊、複雜得多，

much more complicated than the Go board.

It's a lot less visible,

它也遠不如棋盤上那麽黑白分明，

but it's still a decision problem.

但仍然是個判定問題 （Decision Problem）。

And if we think about some of the technologies

如果我們思考一些 即將問世的新科技……

that are coming down the pike ...

Noriko [Arai] mentioned that reading is not yet happening in machines,

新井紀子提到機器仍無法 「閱讀」，

at least with understanding.

至少無法真正理解文本含義。

But that will happen,

但這項能力最終會被機器掌握，

and when that happens,

而當這一切發生時，

very soon afterwards,

不久之後，

machines will have read everything that the human race has ever written.

機器就能讀遍所有人類寫下的東西。

And that will enable machines,

這會讓機器擁有比人類 更深刻的遠見和洞察力。

along with the ability to look further ahead than humans can,

as we've already seen in Go,

就如我們在這場圍棋博弈中所見，

if they also have access to more information,

如果機器能接觸到比人類更多的信息，

they'll be able to make better decisions in the real world than we can.

那機器將能夠在現實世界中 做出比人類更好的決策。

So is that a good thing?

那這會是一件好事嗎？

Well, I hope so.

我當然希望如此。

Our entire civilization, everything that we value,

人類的全部文明， 我們所珍視的一切，

is based on our intelligence.

都是基於我們的智慧。

And if we had access to a lot more intelligence,

如果我們能獲得更強大的智慧，

then there's really no limit to what the human race can do.

那人類將無所不能了。

And I think this could be, as some people have described it,

我在想，到時後就像 一些人所描述的那樣，

the biggest event in human history.

這會是人類歷史上最重要的事件。

So why are people saying things like this,

那為什麽有的人會說出 以下的言論呢？

that AI might spell the end of the human race?

說人工智慧將是人類的末日呢？

Is this a new thing?

這是新鮮事嗎？

Is it just Elon Musk and Bill Gates and Stephen Hawking?

這僅僅只是伊隆馬斯克、比爾蓋茲、 史蒂芬霍金的新發明嗎？

Actually, no. This idea has been around for a while.

實際上不是，這個概念 已經存在很長的時間了。

Here's a quotation:

請看這段話：

"Even if we could keep the machines in a subservient position,

「即便我們能讓機器屈從於我們，

for instance, by turning off the power at strategic moments" --

比如說，在重要時刻關掉它。」

and I'll come back to that "turning off the power" idea later on --

我等會兒會再來討論 「關機」這一話題。

"we should, as a species, feel greatly humbled."

「我們作為人類，仍應懷着謙卑......」

So who said this? This is Alan Turing in 1951.

這段話是誰說的呢？ 是艾倫 · 圖靈在 1951 年說的。

Alan Turing, as you know, is the father of computer science

眾所皆知艾倫 · 圖靈是計算機科學之父，

and in many ways, the father of AI as well.

並且從很多方面來講， 他也是人工智慧之父。

So if we think about this problem,

所以，當我們在思考「創造出 比自己更聰明的物種」這個問題時，

the problem of creating something more intelligent than your own species,

we might call this "the gorilla problem,"

我們不妨將它稱為「大猩猩問題」。

because gorillas' ancestors did this a few million years ago,

因為大猩猩的祖先們 在幾百萬年前就親歷此境，

and now we can ask the gorillas:

我們可以去問大猩猩們：

Was this a good idea?

「這是不是一個好主意？」

So here they are having a meeting to discuss whether it was a good idea,

圖片中，牠們正在開會討論 那麽做是不是一個好主意，

and after a little while, they conclude, no,

過了一會兒，牠們總結出：「不。」

this was a terrible idea.

這是個很爛的主意──

Our species is in dire straits.

作為靈長類的我們正岌岌可危。

In fact, you can see the existential sadness in their eyes.

你可以從牠們的眼神中 看到存亡攸關的憂傷。

(Laughter)

（笑聲）

So this queasy feeling that making something smarter than your own species

「創造出比你自己更聰明的物種 並不是什麽妙計」

is maybe not a good idea --

這種感覺很倒胃口。

what can we do about that?

那我們能做些什麽呢？

Well, really nothing, except stop doing AI,

其實，除非停止人工智慧的研究， 否則束手無策。

and because of all the benefits that I mentioned

因為我所提到的人工智慧的各種裨益，

and because I'm an AI researcher,

也因為我是人工智慧的研究人員，

I'm not having that.

我可不同意就此止步。

I actually want to be able to keep doing AI.

實際上，我想一直研究人工智慧。

So we actually need to nail down the problem a bit more.

所以我們需要更加明確問題所在。

What exactly is the problem?

這個問題到底是什麽呢？

Why is better AI possibly a catastrophe?

為什麽更強大的人工智慧 可能會是個災難呢？

So here's another quotation:

還有一句名言：

"We had better be quite sure that the purpose put into the machine

「我們最好確保我們向機器發出的指令 與我們的真正目的相吻合。」

is the purpose which we really desire."

This was said by Norbert Wiener in 1960,

這句話是諾伯特 · 維納在 1960 年說的，

shortly after he watched one of the very early learning systems

就在他看完一個早期的學習系統 （Learning System）之後。

learn to play checkers better than its creator.

這個系統在學習如何能把 西洋棋下得比發明它的人更好。

But this could equally have been said

但如出一轍的一句話，

by King Midas.

邁達斯國王也說過。

King Midas said, "I want everything I touch to turn to gold,"

他說：「我希望我觸碰的 所有東西都變成金子。」

and he got exactly what he asked for.

結果他真的獲得了點石成金的能力。

That was the purpose that he put into the machine,

可以說，這就是他給機器下的指令。

so to speak,

and then his food and his drink and his relatives turned to gold

結果他的食物、飲料 和家人都變成了金子，

and he died in misery and starvation.

最後他死於痛苦與饑餓當中。

So we'll call this "the King Midas problem"

所以我們把這類問題叫做 「邁達斯國王問題」，

of stating an objective which is not, in fact,

這個比喻是要說明這種 不符合實際需求的 「目的」。

truly aligned with what we want.

In modern terms, we call this "the value alignment problem."

用現代的術語來說，我們把它稱為 「價值取向不一致問題」。

Putting in the wrong objective is not the only part of the problem.

「設錯了目標」不是唯一的問題，

There's another part.

還有其他的。

If you put an objective into a machine,

如果你給機器人設了個目標，

even something as simple as, "Fetch the coffee,"

即使簡單如「去把咖啡端來。」

the machine says to itself,

那機器人會對自己說：

"Well, how might I fail to fetch the coffee?

「什麼會讓我無法去拿咖啡？

Someone might switch me off.

說不定有人會把我關機；

OK, I have to take steps to prevent that.

好，那我要想辦法阻止，

I will disable my 'off' switch.

我得讓我的「關機」開關失效。

I will do anything to defend myself against interference

我得盡一切可能防衛自己，

免得別人干涉我去達成 所被賦予的任務。」

with this objective that I have been given."

So this single-minded pursuit

這種專注的行事，以一種 極端自我保護的模式在執行，

in a very defensive mode of an objective that is, in fact,

not aligned with the true objectives of the human race --

實際上與我們人類 想要的目標並不一致。

that's the problem that we face.

這就是我們面臨的問題。

And in fact, that's the high-value takeaway from this talk.

而這就是這場演講的 核心想法，也是價值所在。

If you want to remember one thing,

如果你想從這場演講中汲取什麽，

it's that you can't fetch the coffee if you're dead.

那你只要記得： 如果死了，就不能端咖啡了。

(Laughter)

（笑聲）

It's very simple. Just remember that. Repeat it to yourself three times a day.

這很簡單，記住就行了， 每天早晚覆誦三遍。

(Laughter)

（笑聲）

And in fact, this is exactly the plot

實際上，這正是電影 《2001太空漫步》的劇情。

of "2001: [A Space Odyssey]"

HAL has an objective, a mission,

HAL 有一個目標，一個任務，

which is not aligned with the objectives of the humans,

但這個目標與人類的目標不一致，

and that leads to this conflict.

最後導致了衝突。

Now fortunately, HAL is not superintelligent.

幸運的是， HAL 並沒有超級智慧，

He's pretty smart, but eventually Dave outwits him

它挺聰明的， 但還是比不過人類戴夫，

and manages to switch him off.

戴夫可以把 HAL 關掉。

But we might not be so lucky.

但我們可能就沒有這麽幸運了。

So what are we going to do?

那我們應該怎麽辦呢？

I'm trying to redefine AI

我想要重新定義人工智慧，

to get away from this classical notion

不再囿於傳統的概念：

of machines that intelligently pursue objectives.

能明智地達成目標的機器。

There are three principles involved.

新的定義涉及三條原則。

The first one is a principle of altruism, if you like,

第一個原則是利他主義原則，

that the robot's only objective

也就是說，機器的唯一目標

is to maximize the realization of human objectives,

就是要最大化地實現 人類的目標、人類的價值。

of human values.

And by values here I don't mean touchy-feely, goody-goody values.

這種價值不是指多愁善感 或者假裝乖巧，

I just mean whatever it is that the human would prefer

而是指人類所嚮往、追求的生活， 無論現狀如何。

their life to be like.

And so this actually violates Asimov's law

事實上，這樣就違反了艾西莫夫定律，

that the robot has to protect its own existence.

定律裡的機器人必須維護自己的生存。

It has no interest in preserving its existence whatsoever.

而在這條原則裡 機器對自身生存與否毫不關心。

The second law is a law of humility, if you like.

第二個原則，不妨稱之為謙遜原則。

And this turns out to be really important to make robots safe.

這一條對製造出安全的機器人十分重要。

It says that the robot does not know

它是指機器人不知道人類的價值是什麽，

what those human values are,

so it has to maximize them, but it doesn't know what they are.

它只知道將該價值最大化， 但卻不知道該價值究竟是什麽。

And that avoids this problem of single-minded pursuit

這就避免了「追求單一目的 而不知變通」的現象。

of an objective.

This uncertainty turns out to be crucial.

這種不確定性就變得很重要了。

Now, in order to be useful to us,

為了對我們有益，

it has to have some idea of what we want.

機械就得大概明白我們想要什麽。

It obtains that information primarily by observation of human choices,

它要獲取這類信息，主要是 透過觀察人類的決策，

so our own choices reveal information

所以我們的決策會揭露 我們生活的意願，

about what it is that we prefer our lives to be like.

So those are the three principles.

所以，這三條原則，

Let's see how that applies to this question of:

讓我們來看看要如何 應用到圖靈所說的問題：

"Can you switch the machine off?" as Turing suggested.

「你能不能將機器關掉？」

So here's a PR2 robot.

這是 PR2 機器人，

This is one that we have in our lab,

這是我們實驗室裡的其中一台，

and it has a big red "off" switch right on the back.

它的背面有一個大大的紅色開關。

The question is: Is it going to let you switch it off?

那問題來了：它會讓你把它關掉嗎？

If we do it the classical way,

如果我們用傳統的定義製造它，

we give it the objective of, "Fetch the coffee, I must fetch the coffee,

我們給它一個「去拿咖啡」的目標， 它會想：「我必須去拿咖啡，

I can't fetch the coffee if I'm dead,"

但如果我死了，就不能拿咖啡了。」

so obviously the PR2 has been listening to my talk,

看來， PR2 聽過我的演講了，

and so it says, therefore, "I must disable my 'off' switch,

因此它說：「我必須讓自己的開關失靈，

and probably taser all the other people in Starbucks

可能還要通過電擊把那些在 星巴克裡干擾我的人都擊暈。」

who might interfere with me."

(Laughter)

（笑聲）

So this seems to be inevitable, right?

這無法避免，對吧？

This kind of failure mode seems to be inevitable,

這種失敗看起來是必然的，

and it follows from having a concrete, definite objective.

因為機器人會遵循一個 十分明確的目標。

So what happens if the machine is uncertain about the objective?

那如果機器對目標 不那麽確定會發生什麽呢？

Well, it reasons in a different way.

那它的思路就不一樣了。

It says, "OK, the human might switch me off,

它會說：「好的，人類可能會把我關掉，

but only if I'm doing something wrong.

但只有我做錯事了，才會把我關掉。

Well, I don't really know what wrong is,

沒錯，我真的不知道什麽才是錯，

but I know that I don't want to do it."

但我知道我不該做錯的事。」

So that's the first and second principles right there.

這就是第一和第二原則。

"So I should let the human switch me off."

「所以我應該讓人類把我關掉。」

And in fact you can calculate the incentive that the robot has

事實上你可以推斷出機器人為了 允許讓人類關掉它所包含的動機，

to allow the human to switch it off,

and it's directly tied to the degree

而且這與根本目標的 不確定性程度直接相關。

of uncertainty about the underlying objective.

And then when the machine is switched off,

當機器被關閉後，

that third principle comes into play.

第三條原則就起作用了。

It learns something about the objectives it should be pursuing,

機器開始學習它應追求的目標，

because it learns that what it did wasn't right.

因為它知道它剛才做的事是不對的。

In fact, we can, with suitable use of Greek symbols,

實際上，我們可以適當地 使用些希臘字母，

as mathematicians usually do,

就像數學家們經常做的那樣，

we can actually prove a theorem

直接證明這一個理論：這樣的 機器人對人類是絕對有利的。

that says that such a robot is provably beneficial to the human.

You are provably better off with a machine that's designed in this way

可以證明如此設計出來的機器人， 對我們的生活是是有益的。

than without it.

So this is a very simple example, but this is the first step

這個例子很簡單，

但它是我們嘗試實現 能與人類和諧共處的 AI 的第一步。

in what we're trying to do with human-compatible AI.

Now, this third principle,

現在來看第三個原則，

I think is the one that you're probably scratching your head over.

我知道各位可能還在為 這一個原則傷腦筋。

You're probably thinking, "Well, you know, I behave badly.

你可能會想：「你懂的， 我行為舉止比較差勁。

I don't want my robot to behave like me.

我的機器人可不能被我帶壞。

I sneak down in the middle of the night and take stuff from the fridge.

我有時後會大半夜偷偷摸摸地 從冰箱裡找東西吃，

I do this and that."

東瞅瞅，西摸摸。」

There's all kinds of things you don't want the robot doing.

有各種各樣的事 你是不希望機器人去做的。

But in fact, it doesn't quite work that way.

但實際上不是那樣。

Just because you behave badly

你行為不檢，

doesn't mean the robot is going to copy your behavior.

不代表機器人就得有樣學樣。

It's going to understand your motivations and maybe help you resist them,

它會去嘗試理解你做事的動機，

而且可能會在合適的情況下 幫助你、制止你。

if appropriate.

但這仍然十分困難。

But it's still difficult.

What we're trying to do, in fact,

實際上，我們是要讓機器

is to allow machines to predict for any person and for any possible life

為任何人、任何一種 可能的生活去預測：

that they could live,

他們更想怎樣？更想要什麽？

and the lives of everybody else:

Which would they prefer?

And there are many, many difficulties involved in doing this;

這涉及到諸多困難，

I don't expect that this is going to get solved very quickly.

我不認為這會很快地就被解決。

The real difficulties, in fact, are us.

實際上，真正的困難是我們自己。

As I have already mentioned, we behave badly.

就像我剛說的那樣， 我們做事不守規矩。

In fact, some of us are downright nasty.

我們當中就有人是非常惡劣的。

Now the robot, as I said, doesn't have to copy the behavior.

如前所說，機器人 未必得要複製那些行為。

The robot does not have any objective of its own.

機器人沒有自己的目標，

It's purely altruistic.

它是完全利他的。

And it's not designed just to satisfy the desires of one person, the user,

它的誕生不僅僅是為了去滿足 某一個人、某一個用戶的欲望，

but in fact it has to respect the preferences of everybody.

而是去尊重所有人的意願。

So it can deal with a certain amount of nastiness,

所以它懂得抵制一些惡劣的行為，

and it can even understand that your nastiness, for example,

它甚至能理解你為什麼惡劣，比如說，

you may take bribes as a passport official

如果你是一個邊境護照官員， 你可能會收取賄賂，

because you need to feed your family and send your kids to school.

因為你得養家、供孩子們上學。

It can understand that; it doesn't mean it's going to steal.

機器人能理解這一點， 但不代表它也會學你偷錢，

In fact, it'll just help you send your kids to school.

它反而會幫助你去供孩子們上學。

We are also computationally limited.

我們的計算能力也是有限的。

Lee Sedol is a brilliant Go player,

李世石是一個傑出的圍棋大師，

but he still lost.

但他還是輸了。

So if we look at his actions, he took an action that lost the game.

如果我們仔細觀察他的棋路， 他下錯了那幾步以致輸棋，

That doesn't mean he wanted to lose.

但這不意味著他想要輸。

So to understand his behavior,

所以要理解他的行為，

we actually have to invert through a model of human cognition

我們得從人類認知的模型回推過來，

that includes our computational limitations -- a very complicated model.

它包含了我們計算能力上的局限，

是一個很覆雜的模型。

But it's still something that we can work on understanding.

但我們仍然可以嘗試去理解。

Probably the most difficult part, from my point of view as an AI researcher,

可能對於我這樣的 AI 研究人員來說，

is the fact that there are lots of us,

最大的困難是，人有很多種，

and so the machine has to somehow trade off, weigh up the preferences

所以機器必須想辦法去協調、 權衡不同人之間的喜好、需求，

of many different people,

and there are different ways to do that.

而要做到這一點有多種不同的方法。

Economists, sociologists, moral philosophers have understood that,

經濟學家、社會學家、 道德哲學家都理解這一點，

and we are actively looking for collaboration.

我們正積極地尋求合作。

Let's have a look and see what happens when you get that wrong.

讓我們來看看，如果我們把這一步 走錯了會怎麽樣。

So you can have a conversation, for example,

比如說，你可能會與你的 人工智慧助理有這樣的對話，

with your intelligent personal assistant

that might be available in a few years' time.

這樣的人工智慧可能幾年內就會出現。

Think of a Siri on steroids.

可以把它想成是強化版的 Siri 。

So Siri says, "Your wife called to remind you about dinner tonight."

Siri 對你說：「你老婆打電話 提醒你別忘了今天的晚宴。」

And of course, you've forgotten. "What? What dinner?

當然你早就忘了這回事：

「什麽？什麽晚宴？你在說什麽？」

What are you talking about?"

"Uh, your 20th anniversary at 7pm."

「呃.....今晚 7 點 慶祝結婚 20 周年。」

"I can't do that. I'm meeting with the secretary-general at 7:30.

「我可去不了， 我晚上 7 點半要見秘書長。

How could this have happened?"

怎麽會這樣呢？」

"Well, I did warn you, but you overrode my recommendation."

「呃，我可是提醒過你的， 但你沒有理會我的建議。」

"Well, what am I going to do? I can't just tell him I'm too busy."

「我該怎麽辦呢？我可不能跟秘書長說 我有事，沒空見他。」

"Don't worry. I arranged for his plane to be delayed."

「別擔心。我已經安排了， 讓他的航班延誤。」

(Laughter)

（笑聲）

"Some kind of computer malfunction."

「用某種電腦故障。」

(Laughter)

（笑聲）

"Really? You can do that?"

「真的嗎？這個你也能做到？」

"He sends his profound apologies

「秘書長很不好意思，跟你道歉，

and looks forward to meeting you for lunch tomorrow."

並邀請你明天中午吃飯。」

(Laughter)

（笑聲）

So the values here -- there's a slight mistake going on.

所以這裡談的價值觀就有點問題了，

This is clearly following my wife's values

這顯然是在遵循我老婆的價值觀，

which is "Happy wife, happy life."

也就是「老婆開心，生活舒心」。

(Laughter)

（笑聲）

It could go the other way.

它也有可能發展成另一種情況。

You could come home after a hard day's work,

你忙碌一天，回到家裏，

and the computer says, "Long day?"

電腦對你說：「今天很忙喔？」

"Yes, I didn't even have time for lunch."

「是啊，我連午飯都沒來得及吃。」

"You must be very hungry."

「那你一定很餓了吧。」

"Starving, yeah. Could you make some dinner?"

「快餓暈了。你能做點晚飯嗎？」

"There's something I need to tell you."

「有一件事我得告訴你。」

(Laughter)

（笑聲）

"There are humans in South Sudan who are in more urgent need than you."

「南蘇丹人民的情況 比你更緊急，更需要照顧。」

(Laughter)

（笑聲）

"So I'm leaving. Make your own dinner."

「所以我要走了。你自己做飯去吧。」

(Laughter)

（笑聲）

So we have to solve these problems,

我們得解決這類的問題，

and I'm looking forward to working on them.

我也很期待能解決這樣的問題。

There are reasons for optimism.

我們有理由感到樂觀。

One reason is,

理由之一是，

there is a massive amount of data.

我們有大量的數據資料。

Because remember -- I said they're going to read everything

記住，我說過機器將能夠 閱讀所有人類寫下來的東西。

the human race has ever written.

而我們寫下的文字大都類似於

Most of what we write about is human beings doing things

「人類做了一些事情 導致其他人對此感到沮喪」。

and other people getting upset about it.

So there's a massive amount of data to learn from.

所以機器可以從 大量的數據中去學習。

There's also a very strong economic incentive

同時從經濟的角度， 我們也有足夠的動機去做好這件事。

to get this right.

So imagine your domestic robot's at home.

想像一下，你家裡有個居家機器人。

You're late from work again and the robot has to feed the kids,

而你又得加班， 機器人得給孩子們做飯，

and the kids are hungry and there's nothing in the fridge.

孩子們很餓， 但冰箱裡什麽都沒有。

And the robot sees the cat.

然後機器人看到了家裡的貓。

(Laughter)

（笑聲）

And the robot hasn't quite learned the human value function properly,

機器人還沒學透人類的價值觀。

so it doesn't understand

所以它不知道， 貓的情感價值大於其營養價值。

the sentimental value of the cat outweighs the nutritional value of the cat.

(Laughter)

（笑聲）

So then what happens?

接下來會發生什麽事？

Well, it happens like this:

頭版頭條可能會是這樣：

"Deranged robot cooks kitty for family dinner."

「瘋狂機器人煮了貓咪當晚餐！」

That one incident would be the end of the domestic robot industry.

這場意外就足以結束 整個居家機器人的產業。

So there's a huge incentive to get this right

所以在我們實現超級 AI 之前， 我們有足夠的動機把它做對做好。

long before we reach superintelligent machines.

So to summarize:

總結來說：

I'm actually trying to change the definition of AI

我事實上想要改變人工智慧的定義，

so that we have provably beneficial machines.

這樣我們就可以製造出 對我們有益無害的機器人。

And the principles are:

這三個原則是：

machines that are altruistic,

機器是利他的，

that want to achieve only our objectives,

只想著實現我們的目標，

but that are uncertain about what those objectives are,

但它不確定我們的目標是什麽，

and will watch all of us

並且它會觀察我們，

to learn more about what it is that we really want.

從中學習我們想要的究竟是什麽。

And hopefully in the process, we will learn to be better people.

希望在這個過程中， 我們也能學會成為更好的人。

Thank you very much.

謝謝大家。

(Applause)

（掌聲）

Chris Anderson: So interesting, Stuart.

克里斯安德森：非常有意思，斯圖爾特。

We're going to stand here a bit because I think they're setting up

趁工作人員為下一位講者佈置的時候，

for our next speaker.

我們先站在這裡聊幾句。

A couple of questions.

我有幾個問題。

So the idea of programming in ignorance seems intuitively really powerful.

將「無知」編寫到程式中， 這種思想真的很有衝擊力。

As you get to superintelligence,

當機器人有超級智慧時，

what's going to stop a robot

還有什麽東西能阻檔機器人閱讀書籍，

reading literature and discovering this idea that knowledge

並了解到：博學比無知要好得多，

is actually better than ignorance

and still just shifting its own goals and rewriting that programming?

進而改變它的目標， 重新編寫自己的程式呢？

Stuart Russell: Yes, so we want it to learn more, as I said,

斯圖爾特拉塞爾：是的， 我們想要它去學習，就像我說的，

about our objectives.

讓機器人學習我們的目標，

It'll only become more certain as it becomes more correct,

只有在理解得越正確的時候， 它們才會更明確我們要的東西，

so the evidence is there

佐證擺在那裡，

and it's going to be designed to interpret it correctly.

並且我們使它能夠正確解讀這些目標。

It will understand, for example, that books are very biased

比如說，它能夠從書中的佐證 判斷出那些富含偏見的書，

in the evidence they contain.

They only talk about kings and princes

像是只講述國王、王子， 和男性精英白人之類的書。

and elite white male people doing stuff.

So it's a complicated problem,

所以這是一個複雜的問題，

but as it learns more about our objectives

但當它更深入地學習我們的目標時，

it will become more and more useful to us.

它會變得越來越有用。

CA: And you couldn't just boil it down to one law,

CA：所以它十分複雜， 遠不足以濃縮成一條法則嗎？

you know, hardwired in:

像是，把這樣的命令燒録進去：

"if any human ever tries to switch me off,

「如果人類想把我關掉，

I comply. I comply."

我要服從。我要服從。」

SR: Absolutely not.

SR：絕對不行。

That would be a terrible idea.

那將是一個很糟糕的主意。

So imagine that you have a self-driving car

試想一下，你有一輛無人駕駛汽車，

and you want to send your five-year-old

你想讓它送你五歲的孩子去幼稚園。

off to preschool.

Do you want your five-year-old to be able to switch off the car

你會希望你五歲的孩子 在汽車運行的過程中將它關閉嗎？

while it's driving along?

應該不會吧。

Probably not.

所以它得理解

So it needs to understand how rational and sensible the person is.

下指令的人有多理智、有多講道理。

The more rational the person,

這個人越理智，

the more willing you are to be switched off.

它就越願意被你關掉。

If the person is completely random or even malicious,

如果這個人是完全思緒混亂 或者甚至是有惡意的，

then you're less willing to be switched off.

那它就不太願意被你關掉了。

CA: All right. Stuart, can I just say,

CA：好吧。斯圖爾特，我得說，

I really, really hope you figure this out for us.

我真的希望你為我們所有人， 找到解決的辦法。

Thank you so much for that talk. That was amazing.

很感謝你的演講。 十分精彩。

SR：謝謝。

CA：謝謝。

SR: Thank you.

(Applause)

（掌聲）