What moral decisions should driverless cars make? | Iyad Rahwan

翻译人员: Yolanda Zhang 校对人员: Bangyou Xiang

Today I'm going to talk about technology and society.

今天我想谈谈技术和社会。

The Department of Transport estimated that last year

据交通部的估算， 在美国，仅去年就有

35,000 people died from traffic crashes in the US alone.

3万5千人死于交通事故。

Worldwide, 1.2 million people die every year in traffic accidents.

而在全世界，每年则有 120万人死于交通事故。

If there was a way we could eliminate 90 percent of those accidents,

如果有一种方法能减少90%的交通事故，

would you support it?

你会支持它吗？

Of course you would.

答案绝对是肯定的。

This is what driverless car technology promises to achieve

这就是无人车技术所承诺实现的目标，

by eliminating the main source of accidents --

通过消除造成事故的主要原因——

human error.

人为过错。

Now picture yourself in a driverless car in the year 2030,

现在想象一下你在2030年中的一天， 坐在一辆无人车里

sitting back and watching this vintage TEDxCambridge video.

悠闲地观看我这个过时的 TEDxCambridge视频。

(Laughter)

（笑声）

All of a sudden,

突然间，

the car experiences mechanical failure and is unable to stop.

车子出现了机械故障，刹车失灵了。

If the car continues,

如果车继续行驶，

it will crash into a bunch of pedestrians crossing the street,

就会冲入正在穿越人行道的人群中，

but the car may swerve,

但是车还可能转向，

hitting one bystander,

撞到路边一个不相干的人，

killing them to save the pedestrians.

用他的生命来换那些行人的生命。

What should the car do, and who should decide?

这辆车该怎么做， 又该是谁来做这个决定呢？

What if instead the car could swerve into a wall,

再如果，这辆车会转向并撞墙，

crashing and killing you, the passenger,

连你在内人车俱毁， 从而挽救其他人的生命，

in order to save those pedestrians?

这会是个更好的选择吗？

This scenario is inspired by the trolley problem,

这个场景假设是受到了 “电车问题”的启发，

which was invented by philosophers a few decades ago

这是几十年前由一群哲学家

to think about ethics.

发起的对道德的拷问。

Now, the way we think about this problem matters.

我们如何思考这个问题非常关键。

We may for example not think about it at all.

我们也许压根儿就 不应该去纠结这个问题。

We may say this scenario is unrealistic,

我们可以辩称这个场景假设不现实，

incredibly unlikely, or just silly.

太不靠谱，简直无聊透顶。

But I think this criticism misses the point

不过我觉得这种批判没有切中要害，

because it takes the scenario too literally.

因为仅仅是停留在了问题表面。

Of course no accident is going to look like this;

当然没有任何事故会出现这种情况；

no accident has two or three options

没有哪个事故会同时出现2-3种选择，

where everybody dies somehow.

而每种选择中都会有人失去生命。

Instead, the car is going to calculate something

相反，车辆自身会做些计算，

like the probability of hitting a certain group of people,

比如撞击一群人的可能性，

if you swerve one direction versus another direction,

如果转向另一个方向，

you might slightly increase the risk to passengers or other drivers

相对于行人来说，你可能略微 增加了乘客，或者其他驾驶员

versus pedestrians.

受伤的可能性。

It's going to be a more complex calculation,

这将会是一个更加复杂的计算，

but it's still going to involve trade-offs,

不过仍然会涉及到某种权衡，

and trade-offs often require ethics.

而这种权衡经常需要做出道德考量。

We might say then, "Well, let's not worry about this.

我们可能会说，“还是别杞人忧天了。

Let's wait until technology is fully ready and 100 percent safe."

不如等到技术完全成熟，能达到 100%安全的时候再用吧。”

Suppose that we can indeed eliminate 90 percent of those accidents,

假设我们的确可以在 未来的10年内消除90%，

or even 99 percent in the next 10 years.

甚至99%的事故。

What if eliminating the last one percent of accidents

如果消除最后这1%的事故

requires 50 more years of research?

却需要再研究50年才能实现呢？

Should we not adopt the technology?

我们是不是应该放弃这项技术了？

That's 60 million people dead in car accidents

按照目前的死亡率计算， 那可还要牺牲

if we maintain the current rate.

6千万人的生命啊。

So the point is,

所以关键在于，

waiting for full safety is also a choice,

等待万无一失的技术也是一种选择，

and it also involves trade-offs.

这里也有权衡的考虑。

People online on social media have been coming up with all sorts of ways

社交媒体上的人们想尽了办法

to not think about this problem.

去回避这个问题。

One person suggested the car should just swerve somehow

有人建议无人车应该把握好角度， 刚好从人群和路边的

in between the passengers --

无辜者之间的缝隙——

(Laughter)

（笑声）

and the bystander.

穿过去。

Of course if that's what the car can do, that's what the car should do.

当然，如果车辆能做到这一点， 毫无疑问就应该这么做。

We're interested in scenarios in which this is not possible.

我们讨论的是无法实现这一点的情况。

And my personal favorite was a suggestion by a blogger

我个人比较赞同一个博主的点子，

to have an eject button in the car that you press --

在车里加装一个弹射按钮——

(Laughter)

（笑声）

just before the car self-destructs.

在车辆自毁前按一下就行了。

(Laughter)

（笑声）

So if we acknowledge that cars will have to make trade-offs on the road,

那么如果我们认同车辆 将不得不在行驶中做出权衡的话，

how do we think about those trade-offs,

我们要如何考量这种权衡

and how do we decide?

并做出决策呢？

Well, maybe we should run a survey to find out what society wants,

也许我们应该做些调查问卷 看看大众是什么想法，

because ultimately,

毕竟最终，

regulations and the law are a reflection of societal values.

规则和法律应该反映社会价值。

So this is what we did.

所以我们做了这么件事儿。

With my collaborators,

跟我的合作者

Jean-François Bonnefon and Azim Shariff,

朗·弗朗索瓦·伯尼夫和 阿米滋·谢里夫一起，

we ran a survey

我们做了一项调查问卷，

in which we presented people with these types of scenarios.

为人们列举了这些假设的场景。

We gave them two options inspired by two philosophers:

受哲学家杰里米·边沁（英国）和 伊曼努尔·康德（德国）的启发，

Jeremy Bentham and Immanuel Kant.

我们给出了两种选择。

Bentham says the car should follow utilitarian ethics:

边沁认为车辆应该 遵循功利主义道德：

it should take the action that will minimize total harm --

它应该采取最小伤害的行动——

even if that action will kill a bystander

即使是以牺牲一个无辜者为代价，

and even if that action will kill the passenger.

即使会令乘客身亡。

Immanuel Kant says the car should follow duty-bound principles,

伊曼努尔·康德则认为 车辆应该遵循义不容辞的原则，

like "Thou shalt not kill."

比如“不可杀人。”

So you should not take an action that explicitly harms a human being,

因此你不应该有意 去伤害一个人，

and you should let the car take its course

应该让车顺其自然行驶，

even if that's going to harm more people.

即使这样会伤害到更多的人。

What do you think?

你会怎么选择？

Bentham or Kant?

支持边沁还是康德？

Here's what we found.

我们得到的结果是这样的。

Most people sided with Bentham.

大部分人赞同边沁的观点。

So it seems that people want cars to be utilitarian,

所以人们似乎希望 车辆是功利主义的，

minimize total harm,

将伤害降到最小，

and that's what we should all do.

我们都应该这么做。

Problem solved.

问题解决了。

But there is a little catch.

不过这里还有个小插曲。

When we asked people whether they would purchase such cars,

当我们问大家他们 会不会买这样一辆车时，

they said, "Absolutely not."

他们不约而同地回答，“绝对不会。”

(Laughter)

（笑声）

They would like to buy cars that protect them at all costs,

他们更希望买能够 不顾一切保障自己安全的车，

but they want everybody else to buy cars that minimize harm.

不过却指望其他人 都买能将伤害降到最低的车。

(Laughter)

（笑声）

We've seen this problem before.

这个问题以前就出现过。

It's called a social dilemma.

叫做社会道德困境。

And to understand the social dilemma,

为了理解这个概念，

we have to go a little bit back in history.

我们要先简单回顾一下历史。

In the 1800s,

在19世纪，

English economist William Forster Lloyd published a pamphlet

英国经济学家威廉·福斯特·劳埃德 出版了一个宣传册，

which describes the following scenario.

里面描述了这样一个场景。

You have a group of farmers --

有一群农场主，

English farmers --

英国农场主，

who are sharing a common land for their sheep to graze.

共同在一片地里放羊。

Now, if each farmer brings a certain number of sheep --

如果每个农场主都 带了一定数量的羊，

let's say three sheep --

比如每家三只，

the land will be rejuvenated,

这片土地上的植被还可以正常再生，

the farmers are happy,

农场主们自然高兴，

the sheep are happy,

羊群也自在逍遥，

everything is good.

一切都相安无事。

Now, if one farmer brings one extra sheep,

如果有一个农场主多放了一只羊，

that farmer will do slightly better, and no one else will be harmed.

他就会获益更多， 不过其他人也都没什么损失。

But if every farmer made that individually rational decision,

但是如果每个农场主 都擅自增加羊的数量，

the land will be overrun, and it will be depleted

土地容量就会饱和，变得不堪重负，

to the detriment of all the farmers,

所有农场主都会受损，

and of course, to the detriment of the sheep.

当然，羊群也会开始挨饿。

We see this problem in many places:

我们在很多场合都见到过这个问题：

in the difficulty of managing overfishing,

比如过度捕捞的困境，

or in reducing carbon emissions to mitigate climate change.

或者应对气候变化的碳减排。

When it comes to the regulation of driverless cars,

而到了无人车的制度问题，

the common land now is basically public safety --

公共土地在这里指的就是公共安全，

that's the common good --

也就是公共利益，

and the farmers are the passengers

而农场主就是乘客，

or the car owners who are choosing to ride in those cars.

或者车主，决定乘车出行的人。

And by making the individually rational choice

通过自作主张把自己的安全凌驾于

of prioritizing their own safety,

其他人的利益之上，

they may collectively be diminishing the common good,

他们可能共同损害了 能将总损失降到最低的

which is minimizing total harm.

公共利益。

It's called the tragedy of the commons,

传统上把这称为

traditionally,

公地悲剧，

but I think in the case of driverless cars,

不过我认为对于无人车来说，

the problem may be a little bit more insidious

问题可能是更深层次的，

because there is not necessarily an individual human being

因为并没有一个人

making those decisions.

去做决策。

So car manufacturers may simply program cars

那么无人车制造商可能会 简单的把行车电脑程序

that will maximize safety for their clients,

设定成最大程度保护车主的安全，

and those cars may learn automatically on their own

而那些车可能会自主学习，

that doing so requires slightly increasing risk for pedestrians.

而这一过程也就会略微增加 对行人的潜在危险。

So to use the sheep metaphor,

跟羊群的比喻类似，

it's like we now have electric sheep that have a mind of their own.

这就好像换成了一批 可以自己思考的机器羊。

(Laughter)

（笑声）

And they may go and graze even if the farmer doesn't know it.

它们可能会自己去吃草， 而农场主对此毫不知情。

So this is what we may call the tragedy of the algorithmic commons,

这就是我们所谓的算法共享悲剧，

and if offers new types of challenges.

这会带来新的类型的挑战。

Typically, traditionally,

通常在传统模式下，

we solve these types of social dilemmas using regulation,

我们可以通过制定规则 来解决这些社会道德困境，

so either governments or communities get together,

政府或者社区共同商讨决定

and they decide collectively what kind of outcome they want

他们能够接受什么样的后果，

and what sort of constraints on individual behavior

以及需要对个人行为施加

they need to implement.

什么形式的限制。

And then using monitoring and enforcement,

通过监管和强制执行，

they can make sure that the public good is preserved.

就可以确定公共利益得到了保障。

So why don't we just,

那么我们为什么不像

as regulators,

立法者那样，

require that all cars minimize harm?

让所有无人车把危险降到最小？

After all, this is what people say they want.

毕竟这是所有人的共同意愿。

And more importantly,

更重要的是，

I can be sure that as an individual,

作为一个个体，我很确定

if I buy a car that may sacrifice me in a very rare case,

如果我买了一辆会在极端情况下 牺牲我的利益的车，

I'm not the only sucker doing that

我不会是唯一一个自残，

while everybody else enjoys unconditional protection.

让其他所有人都受到无条件保护的人。

In our survey, we did ask people whether they would support regulation

在我们的调查问卷中确实 也问了人们，是否会支持立法，

and here's what we found.

调查结果如下。

First of all, people said no to regulation;

首先，人们并不赞同立法，

and second, they said,

其次，他们认为，

"Well if you regulate cars to do this and to minimize total harm,

“如果你们要制定规则保证 这些车造成的损失最小，

I will not buy those cars."

那我肯定不会买。”

So ironically,

讽刺的是，

by regulating cars to minimize harm,

让无人车遵循最小损失原则，

we may actually end up with more harm

我们得到的反而可能是更大的损失，

because people may not opt into the safer technology

因为人们可能放弃 使用这种更安全的技术，

even if it's much safer than human drivers.

即便其安全性远超过人类驾驶员。

I don't have the final answer to this riddle,

对于这场争论 我并没有得到最终的答案，

but I think as a starting point,

不过我认为作为一个开始，

we need society to come together

我们需要团结整个社会

to decide what trade-offs we are comfortable with

来决定哪种折中方案 是大家都可以接受的，

and to come up with ways in which we can enforce those trade-offs.

更要商讨出可以有效推行 这种权衡决策的方法。

As a starting point, my brilliant students,

以此为基础，我的两位出色的学生，

Edmond Awad and Sohan Dsouza,

Edmond Awad和Sohan Dsouza,

built the Moral Machine website,

创建了“道德衡量器”网站，

which generates random scenarios at you --

可以为你设计出随机的场景——

basically a bunch of random dilemmas in a sequence

简单来说就是一系列 按顺序发生的随机困境，

where you have to choose what the car should do in a given scenario.

你需要据此判断无人车应该如何抉择。

And we vary the ages and even the species of the different victims.

我们还对不同的（潜在）受害者 设置了年龄，甚至种族信息。

So far we've collected over five million decisions

目前我们已经搜集到了 超过5百万份决定，

by over one million people worldwide

来自于全世界超过1百万人

from the website.

在网上给出的答案。

And this is helping us form an early picture

这帮助我们 形成了一个概念雏形，

of what trade-offs people are comfortable with

告诉了我们对人们来说 哪些折中方案最适用，

and what matters to them --

他们最在意的是什么——

even across cultures.

甚至跨越了文化障碍。

But more importantly,

不过更重要的是，

doing this exercise is helping people recognize

这项练习能帮助人们认识到

the difficulty of making those choices

做出这些选择有多难，

and that the regulators are tasked with impossible choices.

而立法者更是被要求做出 不现实的选择。

And maybe this will help us as a society understand the kinds of trade-offs

这还可能帮助我们整个社会 去理解那些最终将会被

that will be implemented ultimately in regulation.

纳入法规的折中方案。

And indeed, I was very happy to hear

诚然，我很高兴听到

that the first set of regulations

第一套由交通部

that came from the Department of Transport --

批准的法规——

announced last week --

上周刚刚公布——

included a 15-point checklist for all carmakers to provide,

囊括了需要所有无人车厂商 提供的15点清单，

and number 14 was ethical consideration --

而其中第14点就是道德考量——

how are you going to deal with that.

要如何处理道德困境。

We also have people reflect on their own decisions

我们还通过为大家 提供自己选择的概要，

by giving them summaries of what they chose.

让人们反思自己的决定。

I'll give you one example --

给大家举个例子——

I'm just going to warn you that this is not your typical example,

我要提醒大家 这不是一个典型的例子，

your typical user.

也不是典型的车主。

This is the most sacrificed and the most saved character for this person.

这个人有着最容易牺牲（儿童）， 也最容易被保护的特征（宠物）。

(Laughter)

（笑声）

Some of you may agree with him,

你们中有人可能会赞同他，

or her, we don't know.

或者她，我们并不知道其性别。

But this person also seems to slightly prefer passengers over pedestrians

不过这位调查对象 也似乎更愿意保护乘客，

in their choices

而不是行人，

and is very happy to punish jaywalking.

甚至相当支持严惩横穿马路的行人。

(Laughter)

（笑声）

So let's wrap up.

那么我们来总结一下。

We started with the question -- let's call it the ethical dilemma --

我们由一个问题开始—— 就叫它道德困境问题——

of what the car should do in a specific scenario:

关于在特定条件下 无人车应该如何抉择：

swerve or stay?

转向还是直行？

But then we realized that the problem was a different one.

但之后我们意识到这并不是问题的核心。

It was the problem of how to get society to agree on and enforce

关键的问题在于如何让 大众在他们能够接受的权衡方案中

the trade-offs they're comfortable with.

达成一致并付诸实施。

It's a social dilemma.

这是个社会道德困境。

In the 1940s, Isaac Asimov wrote his famous laws of robotics --

在20世纪40年代， 艾萨克·阿西莫夫 （俄科幻小说家）就写下了他那著名的

the three laws of robotics.

机器人三大法则。

A robot may not harm a human being,

机器人不能伤害人类，

a robot may not disobey a human being,

机器人不能违背人类的命令，

and a robot may not allow itself to come to harm --

机器人不能擅自伤害自己——

in this order of importance.

这是按重要性由高到低排序的。

But after 40 years or so

但是大约40年后，

and after so many stories pushing these laws to the limit,

太多事件不断挑战这些法则的底线，

Asimov introduced the zeroth law

阿西莫夫又引入了第零号法则，

which takes precedence above all,

凌驾于之前所有法则之上，

and it's that a robot may not harm humanity as a whole.

说的是机器人不能伤害人类这个整体。

I don't know what this means in the context of driverless cars

我不太明白在无人车和 其他特殊背景下

or any specific situation,

这句话是什么意思，

and I don't know how we can implement it,

也不清楚我们要如何实践它，

but I think that by recognizing

但我认为通过认识到

that the regulation of driverless cars is not only a technological problem

针对无人车的立法不仅仅是个技术问题，

but also a societal cooperation problem,

还是一个社会合作问题，

I hope that we can at least begin to ask the right questions.

我希望我们至少可以 从提出正确的问题入手。

Thank you.

谢谢大家。

(Applause)

（掌声）