The danger of AI is weirder than you think | Janelle Shane

翻译人员: Archi Xiao 校对人员: Jingdan Niu

So, artificial intelligence

人工智能，

is known for disrupting all kinds of industries.

以能颠覆所有行业广为人知。

What about ice cream?

那冰淇淋呢？

What kind of mind-blowing new flavors could we generate

我们是否能利用先进的人工智能

with the power of an advanced artificial intelligence?

生成令人震惊的新口味呢？

So I teamed up with a group of coders from Kealing Middle School

我和 Kealing 中学的程序员组了个队

to find out the answer to this question.

想要找到答案。

They collected over 1,600 existing ice cream flavors,

他们收集了超过 1600 种 现有的冰淇淋口味，

and together, we fed them to an algorithm to see what it would generate.

接着我们一起把这些口味输入 到算法中看看会有什么结果。

And here are some of the flavors that the AI came up with.

接下来给大家展示一些 人工智能所想到的口味。

[Pumpkin Trash Break]

【南瓜垃圾破裂】

(Laughter)

(笑声)

[Peanut Butter Slime]

【花生酱稀泥】

[Strawberry Cream Disease]

【草莓奶油病】

(Laughter)

(笑声)

These flavors are not delicious, as we might have hoped they would be.

这些口味听起来并没有 我们想象中美味。

So the question is: What happened?

所以问题来了：怎么回事？

What went wrong?

到底哪里出问题了？

Is the AI trying to kill us?

人工智能是想要干掉我们？

Or is it trying to do what we asked, and there was a problem?

还是说它努力想要回应 我们的要求，但是却出问题了？

In movies, when something goes wrong with AI,

在电影中，当人工智能出了错，

it's usually because the AI has decided

通常是因为它们决定

that it doesn't want to obey the humans anymore,

再也不要听从人类的指令，

and it's got its own goals, thank you very much.

它开始有了自己的目标， 不劳驾人类了。

In real life, though, the AI that we actually have

然而现实生活中， 我们现有的人工智能

is not nearly smart enough for that.

还没达到那样的水平。

It has the approximate computing power

它的计算能力大概跟

of an earthworm,

一条小虫子差不多，

or maybe at most a single honeybee,

又或者顶多只是一只小蜜蜂，

and actually, probably maybe less.

实际上可能更弱。

Like, we're constantly learning new things about brains

我们持续从大脑学习到新事物，

that make it clear how much our AIs don't measure up to real brains.

使我们越来越清楚人工智能 与真正的大脑之间的距离。

So today's AI can do a task like identify a pedestrian in a picture,

现在人工智能所达到的大体就是 在图片中识别出行人的程度，

but it doesn't have a concept of what the pedestrian is

但是它并没有 对于行人的概念，

beyond that it's a collection of lines and textures and things.

除此之外它所做的只是 收集线条，质地之类的信息。

It doesn't know what a human actually is.

但是它并不知道人类到底是什么。

So will today's AI do what we ask it to do?

那么现在的人工智能 能否达到我们的要求？

It will if it can,

能力允许的情况下它会，

but it might not do what we actually want.

但是它所做的可能 并不是我们真正想要的。

So let's say that you were trying to get an AI

假设你想要用人工智能

to take this collection of robot parts

利用一堆机器人的零件

and assemble them into some kind of robot to get from Point A to Point B.

组装成一个机器人 从 A 点移动到 B 点。

Now, if you were going to try and solve this problem

如果你想要通过编写 一个传统的计算机程序

by writing a traditional-style computer program,

来解决这个问题，

you would give the program step-by-step instructions

你需要输入一步步的指令，

on how to take these parts,

指示它怎样拿起零件，

how to assemble them into a robot with legs

怎样把这些零件安装成 一个带脚的机器人，

and then how to use those legs to walk to Point B.

以及如何用脚走到 B 点。

But when you're using AI to solve the problem,

但是当你利用人工智能 来解决这个问题的时候，

it goes differently.

情况不太一样。

You don't tell it how to solve the problem,

你不用告诉它 要怎样解决问题，

you just give it the goal,

你只需要给它一个目标，

and it has to figure out for itself via trial and error

它会通过试错 来解决这个问题，

how to reach that goal.

来实现目标。

And it turns out that the way AI tends to solve this particular problem

结果是，貌似人工智能在 解决这一类问题的时候

is by doing this:

会这么做：

it assembles itself into a tower and then falls over

它把自己搭建成 一座塔然后倾倒，

and lands at Point B.

最后在 B 点落下。

And technically, this solves the problem.

从技术的层面上看，的确解决了问题。

Technically, it got to Point B.

从技术上来说的确到达了 B 点。

The danger of AI is not that it's going to rebel against us,

人工智能的危险 不在于它会反抗我们，

it's that it's going to do exactly what we ask it to do.

而是它们会严格按照 我们的要求去做。

So then the trick of working with AI becomes:

所以和人工智能共事的技巧变成了：

How do we set up the problem so that it actually does what we want?

我们该如何设置问题才能让它 做我们真正想做的事？

So this little robot here is being controlled by an AI.

这一台小机器人 由人工智能操控。

The AI came up with a design for the robot legs

人工智能想到了一个 机器人脚部的设计，

and then figured out how to use them to get past all these obstacles.

然后想到了如何 利用它们绕过障碍。

But when David Ha set up this experiment,

但是当大卫·哈 在做这个实验的时候，

he had to set it up with very, very strict limits

他不得不对人工智能 容许搭建起来的脚

on how big the AI was allowed to make the legs,

设立非常、非常严格的限制，

because otherwise ...

不然的话...

(Laughter)

（笑声）

And technically, it got to the end of that obstacle course.

从技术上说，他的确 到达了障碍路线的终点。

So you see how hard it is to get AI to do something as simple as just walk.

现在我们知道了，仅仅是让人工智能 实现简单的行走就有多困难。

So seeing the AI do this, you may say, OK, no fair,

当看到人工智能这么做的时候， 你可能会说，这不公平。

you can't just be a tall tower and fall over,

你不能只是变成 一座塔然后直接倒下，

you have to actually, like, use legs to walk.

你必须得用脚去走路，

And it turns out, that doesn't always work, either.

结果是， 那往往也不行。

This AI's job was to move fast.

这个人工智能的任务是快速移动。

They didn't tell it that it had to run facing forward

他们没有说它应该面向前方奔跑，

or that it couldn't use its arms.

也没有说不能使用它的手臂。

So this is what you get when you train AI to move fast,

这就是当你训练人工智能 快速移动时所能得到的结果，

you get things like somersaulting and silly walks.

你能得到的就是像这样的 空翻或者滑稽漫步。

It's really common.

太常见了。

So is twitching along the floor in a heap.

在地板上扭动前进 也是一样的结果。

(Laughter)

（笑声）

So in my opinion, you know what should have been a whole lot weirder

在我看来，更奇怪的

is the "Terminator" robots.

就是“终结者”机器人。

Hacking "The Matrix" is another thing that AI will do if you give it a chance.

要是有可能的话，人工智能 还真会入侵“黑客帝国"。

So if you train an AI in a simulation,

如果你用仿真环境 训练一个人工智能的话，

it will learn how to do things like hack into the simulation's math errors

它会学习如何入侵到 一个仿真环境中的数学错误里，

and harvest them for energy.

并从中获得能量。

Or it will figure out how to move faster by glitching repeatedly into the floor.

或者会计算出如何通过 不断地在地板上打滑来加快速度。

When you're working with AI,

当你和人工智能一起工作的时候，

it's less like working with another human

不太像是在跟另一个人一起工作，

and a lot more like working with some kind of weird force of nature.

而更像是在和某种 奇怪的自然力量工作。

And it's really easy to accidentally give AI the wrong problem to solve,

一不小心就很容易让人工 智能去破解错误的问题，

and often we don't realize that until something has actually gone wrong.

往往直到出现问题 我们才察觉到不妥。

So here's an experiment I did,

所以我做了这样的一个实验，

where I wanted the AI to copy paint colors,

我想要让人工智能 利用左边的颜色列表

to invent new paint colors,

复制颜料颜色，

given the list like the ones here on the left.

去创造新的颜色。

And here's what the AI actually came up with.

这就是人工智能想到的结果。

[Sindis Poop, Turdly, Suffer, Gray Pubic]

【辛迪斯粪便，如粪球般， 受难，灰色公众】

(Laughter)

（笑声）

So technically,

基本上，

it did what I asked it to.

它达到了我的要求。

I thought I was asking it for, like, nice paint color names,

我以为我给出的要求是， 让它想出美好的颜色名，

but what I was actually asking it to do

但是实际上我让它做的

was just imitate the kinds of letter combinations

只是单纯地模仿 字母的组合，

that it had seen in the original.

那些它在输入中见到的字母组合。

And I didn't tell it anything about what words mean,

而且我并没有告诉它 这些单词的意思是什么，

or that there are maybe some words

或者告诉它也许有些单词

that it should avoid using in these paint colors.

不能用来给颜色命名。

So its entire world is the data that I gave it.

也就是说它的整个世界里 只有我给出的数据。

Like with the ice cream flavors, it doesn't know about anything else.

正如让它发明冰淇淋的口味那样， 它除此之外一无所知。

So it is through the data

也就是通过数据，

that we often accidentally tell AI to do the wrong thing.

我们常常不小心 让人工智能做错事。

This is a fish called a tench.

有一种叫丁鲷的鱼，

And there was a group of researchers

一群研究者尝试过

who trained an AI to identify this tench in pictures.

训练人工智能去 识别图片里的丁鲷。

But then when they asked it

但是当他们试图搞清

what part of the picture it was actually using to identify the fish,

它到底用了图片的 哪个部分去识别这种鱼，

here's what it highlighted.

这是它所显示的部分。

Yes, those are human fingers.

没错，那些是人类的手指。

Why would it be looking for human fingers

为什么它会去识别人类的手指，

if it's trying to identify a fish?

而不是鱼呢？

Well, it turns out that the tench is a trophy fish,

因为丁鲷实际上是一种战利品鱼，

and so in a lot of pictures that the AI had seen of this fish

所以人工智能在被训练时，

during training,

看过的大多数照片中

the fish looked like this.

鱼都长这样。

(Laughter)

（笑声）

And it didn't know that the fingers aren't part of the fish.

而人工智能并不知道原来 手指并不是鱼的一部分。

So you see why it is so hard to design an AI

现在你们应该能想象， 设计一个能真正懂得

that actually can understand what it's looking at.

自己在做什么的人工 智能是多么困难。

And this is why designing the image recognition

这也就是为什么 给无人驾驶汽车

in self-driving cars is so hard,

设计图像识别技术那么困难，

and why so many self-driving car failures

导致无人驾驶失败的原因

are because the AI got confused.

就是，人工智能迷糊了。

I want to talk about an example from 2016.

接下来我想分享一个 发生在 2016 年的故事。

There was a fatal accident when somebody was using Tesla's autopilot AI,

有人在使用特斯拉的 自动驾驶功能时发生了特大事故，

but instead of using it on the highway like it was designed for,

因为这个人工智能是 为上高速路而设计的，

they used it on city streets.

结果车主居然开到市内街道上。

And what happened was,

结果是，

a truck drove out in front of the car and the car failed to brake.

一辆卡车突然出现在轿车前面， 而轿车没有刹车。

Now, the AI definitely was trained to recognize trucks in pictures.

当然这个人工智能受过训练， 能识别图片中的卡车。

But what it looks like happened is

但是当时的情况看起来，

the AI was trained to recognize trucks on highway driving,

人工智能接受的训练是 识别行驶在高速路上的卡车，

where you would expect to see trucks from behind.

理论上你看到的应该是卡车的尾部，

Trucks on the side is not supposed to happen on a highway,

而侧面对着你的卡车 是不会出现在高速路上的，

and so when the AI saw this truck,

所以当人工智能看到这辆卡车的时候，

it looks like the AI recognized it as most likely to be a road sign

可能把卡车认作一个路标，

and therefore, safe to drive underneath.

因此，它判断 从下面开过去是安全的。

Here's an AI misstep from a different field.

接下来是人工智能在 另一个领域的错误示例。

Amazon recently had to give up on a résumé-sorting algorithm

亚马逊最近不得不放弃 一个他们已经开发了一段时间

that they were working on

的简历分类的算法，

when they discovered that the algorithm had learned to discriminate against women.

因为他们发现这个算法 竟然学会了歧视女性。

What happened is they had trained it on example résumés

原因是当他们把过去招聘人员的简历

of people who they had hired in the past.

用作人工智能的训练材料。

And from these examples, the AI learned to avoid the résumés of people

从这些素材中，人工智能学会了 怎样过滤一些应聘者的简历，

who had gone to women's colleges

那些上过女子大学的

or who had the word "women" somewhere in their resume,

或者是那些含有 “女性”字眼的简历，

as in, "women's soccer team" or "Society of Women Engineers."

比如说“女子足球队” 或者“女性工程师学会”。

The AI didn't know that it wasn't supposed to copy this particular thing

人工智能并不知道自己 不应该复制他所见过的

that it had seen the humans do.

人类这种特定的行为。

And technically, it did what they asked it to do.

从技术层面上说， 它的确按要求做到了。

They just accidentally asked it to do the wrong thing.

只是开发者不小心 下错了指令。

And this happens all the time with AI.

这样的情况在人工智能领域屡见不鲜。

AI can be really destructive and not know it.

人工智能破坏力惊人且不自知。

So the AIs that recommend new content in Facebook, in YouTube,

就如用于脸书和油管上 内容推荐的人工智能，

they're optimized to increase the number of clicks and views.

它们被优化以增加 点击量和阅览量。

And unfortunately, one way that they have found of doing this

但是不幸的是，它们实现 目标的其中一个手段，

is to recommend the content of conspiracy theories or bigotry.

就是推荐阴谋论或者偏执内容。

The AIs themselves don't have any concept of what this content actually is,

人工智能本身对这些内容没有概念，

and they don't have any concept of what the consequences might be

也根本不知道推荐这样的内容

of recommending this content.

会产生怎样的后果。

So, when we're working with AI,

所以当我们与人工智能 一起工作的时候，

it's up to us to avoid problems.

我们有责任去规避问题。

And avoiding things going wrong,

规避可能出错的因素，

that may come down to the age-old problem of communication,

这也就带出一个 老生常谈的沟通问题，

where we as humans have to learn how to communicate with AI.

作为人类，我们要学习 怎样和人工智能沟通。

We have to learn what AI is capable of doing and what it's not,

我们必须明白人工智能 能做什么，不能做什么，

and to understand that, with its tiny little worm brain,

要明白，凭它们的那点小脑袋，

AI doesn't really understand what we're trying to ask it to do.

人工智能并不能完全明白 我们想让它们做什么。

So in other words, we have to be prepared to work with AI

换言之，我们必须对与 人工智能共事做好准备，

that's not the super-competent, all-knowing AI of science fiction.

这可不是科幻片里那些 全能全知的人工智能。

We have to be prepared to work with an AI

我们必须准备好跟

that's the one that we actually have in the present day.

眼下存在的人工智能共事。

And present-day AI is plenty weird enough.

现在的人工智能还真的挺奇怪的。

Thank you.

谢谢。

(Applause)

（掌声）