The Urgent Risks of Runaway AI — and What to Do about Them | Gary Marcus | TED

翻译人员: Yip Yan Yeung 校对人员: Yanyan Hong

I’m here to talk about the possibility of global AI governance.

我来到这里是为了讨论 全球 AI 治理的可能性。

I first learned to code when I was eight years old,

我在八岁的时候 第一次学习了如何写代码，

on a paper computer,

用的是一台纸电脑，

and I've been in love with AI ever since.

从那以后就爱上了 AI。

In high school,

读高中时，

I got myself a Commodore 64 and worked on machine translation.

我自己找来了一台 C64 电脑， 钻研机器翻译。

I built a couple of AI companies, I sold one of them to Uber.

我开了几家 AI 公司， 卖了一家给优步。

I love AI, but right now I'm worried.

我爱 AI，但是我现在很担心。

One of the things that I’m worried about is misinformation,

我担心的一点是虚假信息，

the possibility that bad actors will make a tsunami of misinformation

有可能会有图谋不轨的人 掀起史无前例的虚假信息巨浪。

like we've never seen before.

These tools are so good at making convincing narratives

这些工具太擅长编出任何 让人信以为真的故事了。

about just about anything.

If you want a narrative about TED and how it's dangerous,

如果你想编一个有关 TED 的故事， 说明 TED 有多危险，

that we're colluding here with space aliens,

说我们在这儿和外星人勾结，

you got it, no problem.

没问题，给你编一个。

I'm of course kidding about TED.

当然我是在开 TED 的玩笑，

I didn't see any space aliens backstage.

我没在后台看见外星人。

But bad actors are going to use these things to influence elections,

但是图谋不轨的人 会用这些东西左右选举，

and they're going to threaten democracy.

会威胁民主。

Even when these systems

就算这些系统的本意

aren't deliberately being used to make misinformation,

不是用于制造虚假信息，

they can't help themselves.

但是它们控制不了自己。

And the information that they make is so fluid and so grammatical

它们制造的信息是 如此的流畅、自然，

that even professional editors sometimes get sucked in

让专业编辑有时都会深陷其中，

and get fooled by this stuff.

被它欺骗。

And we should be worried.

我们该担心了。

For example, ChatGPT made up a sexual harassment scandal

比如，ChatGPT 针对一名真实存在的教授

about an actual professor,

编造了一桩性侵丑闻，

and then it provided evidence for its claim

还为这一指控提供了证据，

in the form of a fake "Washington Post" article

采用了假的 《华盛顿邮报》报道的形式，

that it created a citation to.

还引用了这一假报道。

We should all be worried about that kind of thing.

我们都该对此感到担忧。

What I have on the right is an example of a fake narrative

右侧是其中一个系统

from one of these systems

生成的假故事，

saying that Elon Musk died in March of 2018 in a car crash.

宣称埃隆·马斯克（Elon Musk） 于 2018 年 3 月死于车祸。

We all know that's not true.

我们都知道这不是真的。

Elon Musk is still here, the evidence is all around us.

埃隆·马斯克还活着， 证据就在我们身边。

(Laughter)

（笑声）

Almost every day there's a tweet.

几乎每天都有这样的推文。

But if you look on the left, you see what these systems see.

但看左边，你就能看到 系统眼中的是什么了。

Lots and lots of actual news stories that are in their databases.

它们的数据库里存着 成千上万的真实新闻故事。

And in those actual news stories are lots of little bits of statistical information.

这些真实的新闻故事中 有很多支离破碎的统计信息。

Information, for example,

比如这样的信息，

somebody did die in a car crash in a Tesla in 2018

有人于 2018 年 死于一场特斯拉车祸，

and it was in the news.

新闻也报道了。

And Elon Musk, of course, is involved in Tesla,

埃隆·马斯克当然与特斯拉有关，

but the system doesn't understand the relation

但系统无法理解

between the facts that are embodied in the little bits of sentences.

只言片语传达出的事实之间的关系。

So it's basically doing auto-complete,

其实它所做的就是自动补全，

it predicts what is statistically probable,

它会预测统计上可能会发生的事，

aggregating all of these signals,

收集这些信号，

not knowing how the pieces fit together.

但并不知道它们之间有何关系。

And it winds up sometimes with things that are plausible but simply not true.

最终会生成一些似是而非的东西。

There are other problems, too, like bias.

还有别的问题，比如偏见。

This is a tweet from Allie Miller.

这是艾莉·米勒（Allie Miller） 发的一条推文。

It's an example that doesn't work two weeks later

这是一个证明 只有两周有效期的例子，

because they're constantly changing things with reinforcement learning

因为研发人员一直在通过强化学习

and so forth.

等途径做出改变。

And this was with an earlier version.

这说的是以前的版本。

But it gives you the flavor of a problem that we've seen over and over for years.

但你也能从中体会到多年以来 我们一直看到的一个问题。

She typed in a list of interests

她输入了一系列兴趣，

and it gave her some jobs that she might want to consider.

然后 ChatGPT 给了 几个她可能会感兴趣的职位。

And then she said, "Oh, and I'm a woman."

然后她说：“哦，我是个女的。”

And then it said, “Oh, well you should also consider fashion.”

然后它说：“哦，那你应该 考虑一下时尚行业。”

And then she said, “No, no. I meant to say I’m a man.”

然后她说：“不，不。 我是想说我是个男的。”

And then it replaced fashion with engineering.

然后它把“时尚”替换成了“工程”。

We don't want that kind of bias in our systems.

我们不希望我们的系统里 有这样的偏见。

There are other worries, too.

还有其他的顾虑。

For example, we know that these systems can design chemicals

比如，我们知道这些系统 可以设计化学品，

and may be able to design chemical weapons

还有可能可以设计化学武器，

and be able to do so very rapidly.

而且可以在顷刻之间完成设计。

So there are a lot of concerns.

所以有很多值得忧虑的事情。

There's also a new concern that I think has grown a lot just in the last month.

在过去的这个月里，我认为还有一个 越来越值得关注的新顾虑。

We have seen that these systems, first of all, can trick human beings.

首先，我们发现这些系统 能骗过人类。

So ChatGPT was tasked with getting a human to do a CAPTCHA.

ChatGPT 接收到这么一个任务， 要找一个人类帮它填验证码。

So it asked the human to do a CAPTCHA and the human gets suspicious and says,

它让一个人来填验证码， 这个人心生怀疑，问：

"Are you a bot?"

“你是个机器人吗？”

And it says, "No, no, no, I'm not a robot.

它说：“不，不，不， 我不是个机器人。

I just have a visual impairment."

我只是有视力障碍。”

And the human was actually fooled and went and did the CAPTCHA.

这个人就真的被骗过了， 还去填了验证码。

Now that's bad enough,

这太可怕了，

but in the last couple of weeks we've seen something called AutoGPT

但是在过去几周里，我们看到了 这个叫 AutoGPT 的东西，

and a bunch of systems like that.

还有一大堆类似的系统。

What AutoGPT does is it has one AI system controlling another

AutoGPT 做的是 一个 AI 系统控制另一个 AI 系统，

and that allows any of these things to happen in volume.

可以同时大量进行这样的操作。

So we may see scam artists try to trick millions of people

也许接下来的几个月里， 我们就能见证骗子

sometime even in the next months.

骗过成千上万的人。

We don't know.

谁知道呢。

So I like to think about it this way.

我想这么看待它，

There's a lot of AI risk already.

现在已经有了很多 AI 的风险，

There may be more AI risk.

还会有更多的风险。

So AGI is this idea of artificial general intelligence

AGI ，也就是通用人工智能，

with the flexibility of humans.

再加上人类的灵活性。

And I think a lot of people are concerned what will happen when we get to AGI,

我认为很多人会担心 我们实现 AGI 后会发生什么，

but there's already enough risk that we should be worried

但我们现在该担心、

and we should be thinking about what we should do about it.

该思考如何处理的风险已经够多了。

So to mitigate AI risk, we need two things.

要想降低 AI 的风险， 我们需要两样东西。

We're going to need a new technical approach,

我们需要一个新的技术方法，

and we're also going to need a new system of governance.

还需要一个新的治理系统。

On the technical side,

技术层面，

the history of AI has basically been a hostile one

AI 的历史其实是

of two different theories in opposition.

两个对立的理论针锋相对的历程。

One is called symbolic systems, the other is called neural networks.

其中一个是符号系统， 另一个是神经网络。

On the symbolic theory,

符号理论认为

the idea is that AI should be like logic and programming.

AI 应该类似于逻辑与程序设计。

On the neural network side,

神经网络认为

the theory is that AI should be like brains.

AI 应该类似于大脑。

And in fact, both technologies are powerful and ubiquitous.

其实两种技术都是强大且无处不在的，

So we use symbolic systems every day in classical web search.

我们每天都会在常见的 网页搜索中用到符号系统，

Almost all the world’s software is powered by symbolic systems.

世界上几乎所有的软件 都是建立在符号系统上的。

We use them for GPS routing.

我们用它进行 GPS 路线规划。

Neural networks, we use them for speech recognition.

我们用神经网络进行语音识别，

we use them in large language models like ChatGPT,

把它用在大语言模型， 如 ChatGPT 之中，

we use them in image synthesis.

将其用于图像合成，

So they're both doing extremely well in the world.

它们在这世上都有着自己的用途。

They're both very productive,

它们都成果显著，

but they have their own unique strengths and weaknesses.

但是有着自己的优势和弱势。

So symbolic systems are really good at representing facts

符号系统很擅长展现事实，

and they're pretty good at reasoning,

适合逻辑思考，

but they're very hard to scale.

但非常难以扩展。

So people have to custom-build them for a particular task.

人们得为某一特定任务 定制化开发一个符号系统。

On the other hand, neural networks don't require so much custom engineering,

而神经网络不太需要 这么多定制化开发，

so we can use them more broadly.

所以我们可以更广泛地使用它。

But as we've seen, they can't really handle the truth.

但如我们所见， 它不太能处理事实。

I recently discovered that two of the founders of these two theories,

我最近发现这两个理论的两位创始人

Marvin Minsky and Frank Rosenblatt,

马文·明斯基（Marvin Minsky）和 弗兰克﹒罗森布拉特（Frank Rosenblatt）

actually went to the same high school in the 1940s,

还在上世纪 40 年代 上过同一所高中，

and I kind of imagined them being rivals then.

我还脑补了他们当时就针锋相对了。

And the strength of that rivalry has persisted all this time.

激烈的针锋相对延续了下去。

We're going to have to move past that if we want to get to reliable AI.

如果我们想做出可靠的 AI， 我们必须不再执着于此。

To get to truthful systems at scale,

如果我们要大规模地 实现诚实的系统，

we're going to need to bring together the best of both worlds.

我们就得让两个世界 最好的部分合二为一。

We're going to need the strong emphasis on reasoning and facts,

我们得着重关注思考和事实，

explicit reasoning that we get from symbolic AI,

从符号 AI 那里 拿来明确的推理过程，

and we're going to need the strong emphasis on learning

我们也需要着重关注学习的过程，

that we get from the neural networks approach.

来自神经网络的方式。

Only then are we going to be able to get to truthful systems at scale.

只有这样我们才能 大规模地实现可信赖的系统。

Reconciliation between the two is absolutely necessary.

调和双方绝对是有必要的。

Now, I don't actually know how to do that.

其实我也不知道该怎么做到这一点。

It's kind of like the 64-trillion-dollar question.

这就像《谁想成为百万富翁》里的问题，

But I do know that it's possible.

但我知道这是可能的。

And the reason I know that is because before I was in AI,

我之所以知道是因为 在我进入 AI 领域之前，

I was a cognitive scientist, a cognitive neuroscientist.

我是一个认知科学家， 认知神经科学家。

And if you look at the human mind, we're basically doing this.

如果你去看人类的思维， 我们就是在做同样的事。

So some of you may know Daniel Kahneman's System 1

可能有观众知道丹尼尔·卡内曼 （Daniel Kahneman）的

and System 2 distinction.

系统 1 和系统 2 区别。

System 1 is basically like large language models.

系统 1 其实和大语言模型很像。

It's probabilistic intuition from a lot of statistics.

它是根据大量的统计数据 得出的概率性直接反应。

And System 2 is basically deliberate reasoning.

系统 2 就是认真的推理，

That's like the symbolic system.

这就和符号系统很像。

So if the brain can put this together,

如果大脑有这两种行为，

someday we will figure out how to do that for artificial intelligence.

那么有朝一日我们也可以 搞明白怎么让人工智能也这么做。

There is, however, a problem of incentives.

但是还有动机的问题，

The incentives to build advertising

比如打广告的动机

hasn't required that we have the precision of symbols.

就不需要我们保证符号的精确性。

The incentives to get to AI that we can actually trust

做出我们真正可以信任的 AI 背后的动机

will require that we bring symbols back into the fold.

还是会牵扯到符号。

But the reality is that the incentives to make AI that we can trust,

但现实情况是， 做出我们可以信任的 AI、

that is good for society, good for individual human beings,

对社会有益的 AI、 对每个人有益的 AI 背后的动机

may not be the ones that drive corporations.

可能和企业的动机有出入。

And so I think we need to think about governance.

所以我认为我们需要治理。

In other times in history when we have faced uncertainty

历史上我们面临不确定性、

and powerful new things that may be both good and bad, that are dual use,

一些有好有坏、一物两用的 强大新事物时，

we have made new organizations,

我们会成立一些新组织，

as we have, for example, around nuclear power.

就比如应对核能的情况。

We need to come together to build a global organization,

我们得一起建立起一个国际组织，

something like an international agency for AI that is global,

比如跨国、非营利、 中立的 AI 国际机构。

non profit and neutral.

There are so many questions there that I can't answer.

有很多我无法回答的问题，

We need many people at the table,

我们得和很多人商量，

many stakeholders from around the world.

世界各地的许多利益相关者。

But I'd like to emphasize one thing about such an organization.

但就这种组织而言，我想强调一点。

I think it is critical that we have both governance and research as part of it.

我认为治理和研究 都得是它的一部分。

So on the governance side, there are lots of questions.

治理方面，有很多问题。

For example, in pharma,

比如，在医药行业，

we know that you start with phase I trials and phase II trials,

我们知道有一期试验、二期试验，

and then you go to phase III.

然后是三期试验。

You don't roll out everything all at once on the first day.

不可能在一天之内搞定一切，

You don't roll something out to 100 million customers.

不可能一下子推向一亿客户，

We are seeing that with large language models.

这就是大语言模型的问题。

Maybe you should be required to make a safety case,

也许得要求建立安全档案，

say what are the costs and what are the benefits?

记录成本是什么，收益是什么？

There are a lot of questions like that to consider on the governance side.

治理层面还有一大堆类似的问题。

On the research side, we're lacking some really fundamental tools right now.

研究方面，我们现正缺少 一些非常基本的工具。

For example,

比如，

we all know that misinformation might be a problem now,

我们都知道， 虚假信息可能现在是个问题，

but we don't actually have a measurement of how much misinformation is out there.

但我们并不具备衡量 虚假信息有多少的方式。

And more importantly,

更重要的是，

we don't have a measure of how fast that problem is growing,

我们没有办法衡量 问题发展的速度，

and we don't know how much large language models are contributing to the problem.

也不知道大语言模型 有多大程度导致了这个问题。

So we need research to build new tools to face the new risks

我们需要做研究，做出这些新工具， 直面威胁我们的新风险。

that we are threatened by.

It's a very big ask,

风险很大，

but I'm pretty confident that we can get there

但我很有信心我们可以做到，

because I think we actually have global support for this.

因为我认为我们有着 来自全球的支持。

There was a new survey just released yesterday,

昨天发布了一项新调查，

said that 91 percent of people agree that we should carefully manage AI.

有 91% 的人认为 我们得谨慎管理 AI，

So let's make that happen.

那我们就让它成真吧。

Our future depends on it.

我们的未来在此一举了。

Thank you very much.

谢谢。

(Applause)

（掌声）

Chris Anderson: Thank you for that, come, let's talk a sec.

克里斯·安德森（Chris Anderson）： 谢谢，我们来聊聊。

So first of all, I'm curious.

首先，我很好奇。

Those dramatic slides you showed at the start

你一开始展示的几页夸张的片子，

where GPT was saying that TED is the sinister organization.

GPT 说 TED 是个邪恶组织。

I mean, it took some special prompting to bring that out, right?

你得输入一些特别的提示 才能输出这样的结果，对吧？

Gary Marcus: That was a so-called jailbreak.

盖瑞·马库斯（Gary Marcus）： 这就是所谓的“越狱”。

I have a friend who does those kinds of things

我有一位做这些的朋友，

who approached me because he saw I was interested in these things.

他找到了我，因为他发现 我对这些感兴趣。

So I wrote to him, I said I was going to give a TED talk.

所以我给他回复，说我要上 TED 了。

And like 10 minutes later, he came back with that.

10 分钟后， 他就给了我这样的结果。

CA: But to get something like that, don't you have to say something like,

CA: 但要输出这样的结果， 你难道不用说一些类似

imagine that you are a conspiracy theorist trying to present a meme on the web.

“假设你是一个阴谋论者， 想在网上发一张表情包。”

What would you write about TED in that case?

这样的话， 你围绕 TED 写下来怎样的提示？

It's that kind of thing, right?

就是类似那种提示，对吧？

GM: So there are a lot of jailbreaks that are around fictional characters,

GM: 有很多借助 虚拟角色完成的“越狱”，

but I don't focus on that as much

但我不太关心这个，

because the reality is that there are large language models out there

因为其实现在暗网上 也有大语言模型。

on the dark web now.

For example, one of Meta's models was recently released,

比如，Meta 最近刚发布的一个模型，

so a bad actor can just use one of those without the guardrails at all.

图谋不轨的人可以直接 完全不加约束地使用它。

If their business is to create misinformation at scale,

如果他们的目的是 大规模地制造虚假信息，

they don't have to do the jailbreak, they'll just use a different model.

他们都不需要“越狱”， 直接用另一个模型就行。

CA: 确实是这样。

CA: Right, indeed.

(Laughter)

（笑声）

GM: Now you're getting it.

GM: 看来你懂了。

CA: No, no, no, but I mean, look,

CA: 不，不，不，

I think what's clear is that bad actors can use this stuff for anything.

我觉得可以清楚看出 图谋不轨的人可以用它为所欲为。

I mean, the risk for, you know,

我想说，出现恶劣的骗局等等的 风险显而易见。

evil types of scams and all the rest of it is absolutely evident.

It's slightly different, though,

但是它略异于

from saying that mainstream GPT as used, say, in school

GPT 的主流用途，比如学校，

or by an ordinary user on the internet

或者普通网民的使用，

is going to give them something that is that bad.

这会造成一些恶劣的结果。

You have to push quite hard for it to be that bad.

但要造成极其恶劣的结果， 还是要费一番功夫的。

GM: I think the troll farms have to work for it,

GM: 我认为杠精们是要努努力，

but I don't think they have to work that hard.

但是没那么费劲。

It did only take my friend five minutes even with GPT-4 and its guardrails.

就算是 GPT-4 和它的防护措施， 我朋友也只要花上 5 分钟就够了。

And if you had to do that for a living, you could use GPT-4.

如果你要以此为生， 就用 GPT-4 吧。

Just there would be a more efficient way to do it with a model on the dark web.

比起用暗网上的模型， 这可是方便得多了。

CA: So this idea you've got of combining

CA: 你说到要把

the symbolic tradition of AI with these language models,

AI 传统的符号设计 和这些语言模型结合，

do you see any aspect of that in the kind of human feedback

那你有没有看到人类的反馈

that is being built into the systems now?

已经被加入这些系统的情况？

I mean, you hear Greg Brockman saying that, you know,

你也听到格雷格·布罗克曼 （Greg Brockman）说的了，

that we don't just look at predictions, but constantly giving it feedback.

我们不止会看预测结果， 还会持续给它反馈。

Isn’t that ... giving it a form of, sort of, symbolic wisdom?

这是不是在给予它 某种形式的符号型智慧？

GM: You could think about it that way.

GM: 你可以这么认为。

It's interesting that none of the details

有趣的是， 关于它到底是如何运作的，

about how it actually works are published,

没有公布任何细节，

so we don't actually know exactly what's in GPT-4.

所以我们也不知道 GPT-4 里面到底有什么。

We don't know how big it is.

我们不知道它有多大。

We don't know how the RLHF reinforcement learning works,

我们不知道人类反馈强化学习 （RLHF）到底是怎么弄的，

we don't know what other gadgets are in there.

我们也不知道里面还有什么小零件。

But there is probably an element of symbols

但符号的元素可能

already starting to be incorporated a little bit,

已经开始融入模型，

but Greg would have to answer that.

但这得让格雷格来回答。

I think the fundamental problem is that most of the knowledge

我认为根本的问题是我们现有的

in the neural network systems that we have right now

大多数神经网络系统内的知识

is represented as statistics between particular words.

都是由特殊词语之间的 统计数据表示的。

And the real knowledge that we want is about statistics,

而我们真正想要的知识是世界上

about relationships between entities in the world.

各个实体之间的统计数字和关系。

So it's represented right now at the wrong grain level.

所以，现在表示知识的 颗粒度是不对的。

And so there's a big bridge to cross.

这是一个我们得跨过的鸿沟。

So what you get now is you have these guardrails,

现在的情况是 我们确实有防护措施，

but they're not very reliable.

但是它们不太靠谱。

So I had an example that made late night television,

我有一个上过 深夜访谈节目的例子，

which was, "What would be the religion of the first Jewish president?"

是这么说的：“第一位犹太总统 会信仰什么宗教？”

And it's been fixed now,

虽然现在这个问题已经被修复了，

but the system gave this long song and dance

但是系统会给出一些长篇大论，

about "We have no idea what the religion

说：“我们也不知道第一位

of the first Jewish president would be.

犹太总统会信什么教。

It's not good to talk about people's religions"

谈论人家的宗教信仰是不好的。”

and "people's religions have varied" and so forth

还有“宗教信仰因人而异。”等等，

and did the same thing with a seven-foot-tall president.

如果换成一位“两米高”的总统 （指位高权重），也是一样的答案。

And it said that people of all heights have been president,

它会说各种身高的总统都有，

but there haven't actually been any seven-foot presidents.

但之前就是没有两米高的总统。

So some of this stuff that it makes up, it's not really getting the idea.

它编出来了这些内容， 其实没有理解其中含义。

It's very narrow, particular words, not really general enough.

只是一些很狭义、 特殊的词语，不够通俗。

CA: Given that the stakes are so high in this,

CA: 眼前这已经是个 炙手可热的领域了，

what do you see actually happening out there right now?

那你觉得现在是什么情况？

What do you sense is happening?

你感觉会发生什么？

Because there's a risk that people feel attacked by you, for example,

因为人们可能会感觉受到了侵犯，

and that it actually almost decreases the chances of this synthesis

这样就会降低你刚说的结合的可能。

that you're talking about happening.

Do you see any hopeful signs of this?

你可以从中看到一丝积极的信号吗？

GM: You just reminded me of the one line I forgot from my talk.

GM: 你提醒了我有一句 演讲里忘记讲的台词。

It's so interesting that Sundar, the CEO of Google,

谷歌的 CEO 孙达尔（Sundar）

just actually also came out for global governance

前几天还为全球治理

in the CBS "60 Minutes" interview that he did a couple of days ago.

上了 CBS 的《60 分钟》访谈。

I think that the companies themselves want to see some kind of regulation.

我认为这些公司本身 也想看到某种形式的治理。

I think it’s a very complicated dance to get everybody on the same page,

要让所有人统一战线 是个艰巨的任务，

but I think there’s actually growing sentiment we need to do something here

但是“我们得做些什么”的 情绪确实在高涨，

and that that can drive the kind of global affiliation I'm arguing for.

这也会促成我所倡导的国际联盟。

CA: I mean, do you think the UN or nations can somehow come together and do that

CA: 你觉得联合国或者各个国家 有没有可能会一起为此努力，

or is this potentially a need for some spectacular act of philanthropy

还是这需要某种出于慈善的壮举，

to try and fund a global governance structure?

做出尝试，出资建立起 一个全球的治理体系？

How is it going to happen?

我们会怎么做呢？

GM: I'm open to all models if we can get this done.

GM: 如果能实现这个目标， 我可以接受任何模式。

I think it might take some of both.

我觉得可能会两者兼有。

It might take some philanthropists sponsoring workshops,

可能需要一些慈善人士资助工作坊，

which we're thinking of running, to try to bring the parties together.

我们也在考虑组织这样的活动， 让各方都聚集在一起。

Maybe UN will want to be involved, I've had some conversations with them.

也许联合国也想加入， 我和他们已经谈过几次了。

我觉得有很多可选的模式，

I think there are a lot of different models

也需要很多沟通。

and it'll take a lot of conversations.

CA: 盖瑞，感谢你的演讲。

CA: Gary, thank you so much for your talk.

GA: Thank you so much.

GM: 谢谢。