The Urgent Risks of Runaway AI — and What to Do about Them | Gary Marcus | TED

譯者: 麗玲 辛 審譯者: Helen Chang

I’m here to talk about the possibility of global AI governance.

我來這裡討論 全球 AI 治理的可能性。

I first learned to code when I was eight years old,

我八歲時初次學會寫程式， 用的是紙上電腦，

on a paper computer,

and I've been in love with AI ever since.

我從此就愛上了 AI。

In high school,

高中時，我買了台 C64 電腦 來研究機器翻譯。

I got myself a Commodore 64 and worked on machine translation.

I built a couple of AI companies, I sold one of them to Uber.

我開了幾家 AI 公司， 賣了一家給優步，

I love AI, but right now I'm worried.

我愛 AI ，但現在我很擔憂。

One of the things that I’m worried about is misinformation,

我的擔憂之一是錯誤資訊，

圖謀不軌的人可能製造 史無前例的海量假消息。

the possibility that bad actors will make a tsunami of misinformation

like we've never seen before.

These tools are so good at making convincing narratives

這些工具十分擅長 編造令人信服的故事，

about just about anything.

任何主題都可能。

If you want a narrative about TED and how it's dangerous,

如果你想要編個故事， 說 TED 很危險，

that we're colluding here with space aliens,

說我們在這裡跟外星人勾結，

you got it, no problem.

沒問題，馬上編給你。

I'm of course kidding about TED.

TED 的事當然是開玩笑的。

I didn't see any space aliens backstage.

我在後台沒看到任何外星人。

But bad actors are going to use these things to influence elections,

但圖謀不軌的人 會用這些訊息左右選舉，

and they're going to threaten democracy.

這會威脅到民主。

Even when these systems

即使這些系統的本意 並不是要用來製造錯誤資訊，

aren't deliberately being used to make misinformation,

they can't help themselves.

它們也無從控制。

And the information that they make is so fluid and so grammatical

這些系統所製造出的內容 相當流暢、自然，

that even professional editors sometimes get sucked in

就連專業的編輯也有可能 被這些東西所吸引和愚弄。

and get fooled by this stuff.

And we should be worried.

可能需要一些慈善家來贊助 我們想舉辦的研討會，

For example, ChatGPT made up a sexual harassment scandal

比如，ChatGPT 捏造了 一起性騷擾醜聞，

about an actual professor,

主角是位教授，真有其人，

and then it provided evidence for its claim

然後它以一篇虛假的 《華盛頓郵報》報導的形式

in the form of a fake "Washington Post" article

為其聲明提供了證據， 並引用了該報導。

that it created a citation to.

We should all be worried about that kind of thing.

我們都應為這樣的事擔憂。

What I have on the right is an example of a fake narrative

在螢幕右邊，

是這類系統所製造的 假新聞的一個例子，

from one of these systems

saying that Elon Musk died in March of 2018 in a car crash.

假新聞說伊隆‧馬斯克 在 2018 因為車禍身亡。

We all know that's not true.

我們都知道這不是真的， 馬斯克仍活著，證據就在我們身邊。

Elon Musk is still here, the evidence is all around us.

(Laughter)

（笑聲）

Almost every day there's a tweet.

幾乎每天都有推文。

But if you look on the left, you see what these systems see.

但螢幕左邊呈現的是系統所看見的，

Lots and lots of actual news stories that are in their databases.

這些是它們資料庫中 成千上萬的真實新聞報導。

And in those actual news stories are lots of little bits of statistical information.

在那些真實的新聞報導中， 有許多統計資訊的小片段。

Information, for example,

舉例來說，

somebody did die in a car crash in a Tesla in 2018

2018 年確實有人 在特斯拉的車禍中死亡，

and it was in the news.

新聞有報導。

And Elon Musk, of course, is involved in Tesla,

當然，伊隆‧馬斯克和特斯拉有關，

but the system doesn't understand the relation

但系統無法理解這些在片段句子中的 事實之間有什麼關係。

between the facts that are embodied in the little bits of sentences.

So it's basically doing auto-complete,

所以基本上它在進行自動補全，

it predicts what is statistically probable,

它會預測在統計上可能發生的事，

aggregating all of these signals,

整合所有這些信號，

not knowing how the pieces fit together.

但並不知道這些拼圖該如何拼起來。

And it winds up sometimes with things that are plausible but simply not true.

有時就會出現一些看似有理 但就不是事實的情況。

There are other problems, too, like bias.

還有其他的問題，比如偏見。

This is a tweet from Allie Miller.

這是艾莉‧米勒發的推文。

It's an example that doesn't work two weeks later

這個例子兩週後就會失效，

because they're constantly changing things with reinforcement learning

因為研發人員經常 用強化學習等方式做改善。

and so forth.

這是比較早的版本。

And this was with an earlier version.

But it gives you the flavor of a problem that we've seen over and over for years.

但這能讓各位感受一下我們 年復一年不斷看到的問題。

She typed in a list of interests

她輸入一張興趣清單，

and it gave her some jobs that she might want to consider.

ChatGPT 告訴她 可以考慮哪些工作。

And then she said, "Oh, and I'm a woman."

接著她說：「喔，我是女性。」

And then it said, “Oh, well you should also consider fashion.”

接著它說：「喔，那你 也可以考慮時尚業。」

And then she said, “No, no. I meant to say I’m a man.”

接著她說：「不，不， 說錯了，我是男性。」

And then it replaced fashion with engineering.

接著它就把時尚改為工程。

We don't want that kind of bias in our systems.

我們不希望我們的系統出現這種偏見。

There are other worries, too.

還有其他的顧慮，比如，

For example, we know that these systems can design chemicals

我們知道這些系統能設計化學物， 也許能設計化學武器，

and may be able to design chemical weapons

and be able to do so very rapidly.

而且頃刻之間就能完成設計。

So there are a lot of concerns.

因此，有很多顧慮。

There's also a new concern that I think has grown a lot just in the last month.

在過去一個月裏，我認為 還有件愈來愈值得關注的事。

We have seen that these systems, first of all, can trick human beings.

首先，我們發現這些系統會騙人。

So ChatGPT was tasked with getting a human to do a CAPTCHA.

是這樣，ChatGPT 接到任務， 要找一個人幫它輸入人機驗證碼，

So it asked the human to do a CAPTCHA and the human gets suspicious and says,

當它請那個人輸入驗證碼時，

那個人起了疑心，說： 「你是機器人嗎？」

"Are you a bot?"

And it says, "No, no, no, I'm not a robot.

它說：「不，我不是機器人， 我只是有視力障礙。」

I just have a visual impairment."

And the human was actually fooled and went and did the CAPTCHA.

那個人真的被騙，還去輸入驗證碼。

Now that's bad enough,

那已經夠糟了，但在過去幾週，

but in the last couple of weeks we've seen something called AutoGPT

我們看到了 AutoGPT 以及一堆類似的系統。

and a bunch of systems like that.

AutoGPT 是以一個 AI 系統控制另一個 AI 系統，

What AutoGPT does is it has one AI system controlling another

and that allows any of these things to happen in volume.

可以同時大量進行這樣的操作。

So we may see scam artists try to trick millions of people

也許接下來的幾個月，我們會看到 詐騙高手試圖欺騙數百萬人。

sometime even in the next months.

We don't know.

很難說。

So I like to think about it this way.

我是這麼看這個現象：

There's a lot of AI risk already.

已經有很多 AI 風險存在。

There may be more AI risk.

可能還有更多風險。

So AGI is this idea of artificial general intelligence

AGI 這個概念是人工通用智慧

with the flexibility of humans.

加上人類的靈活性。

And I think a lot of people are concerned what will happen when we get to AGI,

我想有很多人會擔心， 當 AGI 出現時會如何，

but there's already enough risk that we should be worried

但現有的風險就已經夠我們擔心了，

and we should be thinking about what we should do about it.

我們該思考要如何因應。

要降低 AI 風險， 我們需要兩樣東西

So to mitigate AI risk, we need two things.

We're going to need a new technical approach,

我們需要一種新的技術方法，

and we're also going to need a new system of governance.

也需要一種新的管理體制。

On the technical side,

在技術面上，

the history of AI has basically been a hostile one

AI 發展史基本上一直是

兩種不同理論針鋒相對的歷程。

of two different theories in opposition.

One is called symbolic systems, the other is called neural networks.

一種是符號系統， 另一種是神經網絡。

On the symbolic theory,

符號理論認為 AI 應該要像 邏輯及程式設計一樣。

the idea is that AI should be like logic and programming.

On the neural network side,

神經網絡理論則主張 AI 應該要像大腦。

the theory is that AI should be like brains.

And in fact, both technologies are powerful and ubiquitous.

事實上，這兩種技術 都很強大且處處可見。

So we use symbolic systems every day in classical web search.

我們每天常做的網頁搜尋 就是使用符號系統。

Almost all the world’s software is powered by symbolic systems.

幾乎全世界的軟體 都是由符號系統所驅動。

We use them for GPS routing.

我們用符號系統做 GPS 路線規劃。

Neural networks, we use them for speech recognition.

神經網絡則是用來做語音辨識，

we use them in large language models like ChatGPT,

應用在大型語言模型， 如 ChatGPT，

we use them in image synthesis.

還有影像合成。

So they're both doing extremely well in the world.

兩種技術都有出色的表現，富有成效，

They're both very productive,

but they have their own unique strengths and weaknesses.

但各有獨特的優點和缺點。

So symbolic systems are really good at representing facts

符號系統很擅長描繪事實，

and they're pretty good at reasoning,

也很會推理，

but they're very hard to scale.

但規模很難擴大。

So people have to custom-build them for a particular task.

所以必須針對特定的任務 來客製開發一個符號系統。

On the other hand, neural networks don't require so much custom engineering,

另一方面，神經網絡不需要 這麼多客製化工程，

so we can use them more broadly.

所以可以更廣泛運用它們。

But as we've seen, they can't really handle the truth.

但如剛才所見， 它們不太能處理事實。

I recently discovered that two of the founders of these two theories,

我最近發現，

這兩個理論的兩位創始者， 馬文‧明斯基和法蘭克‧羅森布萊特，

Marvin Minsky and Frank Rosenblatt,

actually went to the same high school in the 1940s,

在 1940 年代就讀同一所高中，

and I kind of imagined them being rivals then.

我想像他們在當時是競爭對手。

And the strength of that rivalry has persisted all this time.

而那種競爭的力量一直持續至今。

We're going to have to move past that if we want to get to reliable AI.

但如果要發展出可靠的 AI， 我們就得超越那種對立。

To get to truthful systems at scale,

要建立規模大又能講實話的系統，

we're going to need to bring together the best of both worlds.

我們得把兩個領域的 最佳優點結合起來。

我們需要著重邏輯及事實， 明確的推理強項，

We're going to need the strong emphasis on reasoning and facts,

explicit reasoning that we get from symbolic AI,

and we're going to need the strong emphasis on learning

我們也需要著重學習過程，

that we get from the neural networks approach.

來自於神經網絡的方法。

Only then are we going to be able to get to truthful systems at scale.

唯有這樣，我們才能建立 大規模而可信賴的系統，

Reconciliation between the two is absolutely necessary.

這兩個領域的和解是絕對必要的。

Now, I don't actually know how to do that.

但我並不知道如何做到。

It's kind of like the 64-trillion-dollar question.

這是個非常重要又有價值的問題。

But I do know that it's possible.

但我知道這是可能的，

And the reason I know that is because before I was in AI,

因為在進入 AI 領域之前， 我是認知科學家，認知神經科學家。

I was a cognitive scientist, a cognitive neuroscientist.

And if you look at the human mind, we're basically doing this.

如果你看看人類的思維， 我們現在就是在做同樣的事。

So some of you may know Daniel Kahneman's System 1

你可能知道丹尼爾·康納曼的 系統一和系統二的區別。

and System 2 distinction.

System 1 is basically like large language models.

基本上，系統一 就像是大型語言模型，

It's probabilistic intuition from a lot of statistics.

根據大量統計數據， 得出的概率直覺。

And System 2 is basically deliberate reasoning.

系統二基本上是深思熟慮的推理，

That's like the symbolic system.

就像符號系統。

既然大腦能結合兩者，

So if the brain can put this together,

someday we will figure out how to do that for artificial intelligence.

有一天我們也會弄清楚 怎麼讓 AI 做到這一點。

There is, however, a problem of incentives.

然而，這還有誘因的問題。

The incentives to build advertising

打造廣告並不需要 保證精確的符號。

hasn't required that we have the precision of symbols.

The incentives to get to AI that we can actually trust

但打造可信賴的 AI 就需要把符號系統納入其中。

will require that we bring symbols back into the fold.

But the reality is that the incentives to make AI that we can trust,

但現實是，促使我們去打造 對社會及每個人都有益的可信 AI，

that is good for society, good for individual human beings,

may not be the ones that drive corporations.

或許不是能夠驅動企業的誘因。

And so I think we need to think about governance.

因此我認為我們得考量治理層面。

In other times in history when we have faced uncertainty

歷史上，當我們面對不確定性高，

and powerful new things that may be both good and bad, that are dual use,

可好可壞、具雙重用途的 強大新事物時，

we have made new organizations,

我們會成立新組織，

as we have, for example, around nuclear power.

比如針對核能，我們就有這麼做。

We need to come together to build a global organization,

我們得要團結起來， 建立一個全球組織，

something like an international agency for AI that is global,

跨國、非營利、中立的 AI 國際機構。

non profit and neutral.

There are so many questions there that I can't answer.

還有太多問題，我無法回答，

We need many people at the table,

因此需要許多人參與討論， 來自世界各地的利益相關者。

many stakeholders from around the world.

But I'd like to emphasize one thing about such an organization.

但，針對這種組織，我要強調一點。

I think it is critical that we have both governance and research as part of it.

我認為很重要的一點是 這組織得同時涵蓋治理以及研究。

So on the governance side, there are lots of questions.

在治理面，有很多問題。

比如，在製藥業，

For example, in pharma,

we know that you start with phase I trials and phase II trials,

我們知道要先做 第一期和第二期試驗，

and then you go to phase III.

接著再做第三期， 不能在第一天一起進行。

You don't roll out everything all at once on the first day.

You don't roll something out to 100 million customers.

你不能把產品一下子 就推出給一億個客戶。

We are seeing that with large language models.

大型語言模型就是如此。 也許應該要求建立安全案例，

Maybe you should be required to make a safety case,

say what are the costs and what are the benefits?

了解成本是多少，收益是什麼？

在治理面，要考量很多像這樣的問題。

There are a lot of questions like that to consider on the governance side.

On the research side, we're lacking some really fundamental tools right now.

在研究方面，我們現在 缺乏一些十分基本的工具。

For example,

例如，

we all know that misinformation might be a problem now,

我們都知道錯誤資訊現在是個問題，

but we don't actually have a measurement of how much misinformation is out there.

但實際上我們無法衡量 在網路上到底有多少錯誤資訊。

更重要的是，我們無法測量 這個問題發展的速度，

And more importantly,

we don't have a measure of how fast that problem is growing,

and we don't know how much large language models are contributing to the problem.

也不知道大型語言模型 有多少程度導致了這個問題。

因此，我們需要進行研究， 構建新工具，

So we need research to build new tools to face the new risks

應對我們面臨的新風險。

that we are threatened by.

It's a very big ask,

這是一個重大的任務，

but I'm pretty confident that we can get there

但我非常有信心我們能夠做到這點，

because I think we actually have global support for this.

因為我認為我們已得到全球的支持。

There was a new survey just released yesterday,

昨天剛剛發布的一項新調查顯示，

said that 91 percent of people agree that we should carefully manage AI.

91% 的人同意 我們應該謹慎管理 AI。

So let's make that happen.

因此，讓我們實現這一目標。

Our future depends on it.

我們的未來取決於此。

Thank you very much.

非常感謝。

(Applause)

（掌聲）

Chris Anderson: Thank you for that, come, let's talk a sec.

克里斯·安德森：謝謝你， 來，我們聊聊。

So first of all, I'm curious.

首先，我很好奇。

Those dramatic slides you showed at the start

你在一開始展示的那些很誇張的簡報，

where GPT was saying that TED is the sinister organization.

其中 GPT 說 TED 是邪惡的組織。

I mean, it took some special prompting to bring that out, right?

需要一些特別的提示詞， 它才會這麼說吧？

Gary Marcus: That was a so-called jailbreak.

加里·馬庫斯：那就是所謂的越獄。

I have a friend who does those kinds of things

我有個朋友在做這類研究，

who approached me because he saw I was interested in these things.

他連絡我，因為他發現 我對這些事情很感興趣。

So I wrote to him, I said I was going to give a TED talk.

所以我寫信告訴他， 我要去 TED 演講。

And like 10 minutes later, he came back with that.

大約 10 分鐘後， 他就給了我這個內容。

CA: But to get something like that, don't you have to say something like,

CA：但是要得到這樣的結果， 難道不是要先說：

「假設你是個陰謀論者， 要在網頁上展示迷因梗圖。

imagine that you are a conspiracy theorist trying to present a meme on the web.

What would you write about TED in that case?

在這種情況下，關於 TED， 你會寫什麼？」類似這種提示？

It's that kind of thing, right?

GM: So there are a lot of jailbreaks that are around fictional characters,

GM：現在有很多 借助虛構人物的越獄，

but I don't focus on that as much

我倒沒有那麼關注，

because the reality is that there are large language models out there

因為其實，現在暗網上 就有大量的語言模型。

on the dark web now.

For example, one of Meta's models was recently released,

例如，最近 Meta 的 一個模型被釋出，

so a bad actor can just use one of those without the guardrails at all.

圖謀不軌的人就可以 自由使用其中一個模型。

If their business is to create misinformation at scale,

如果他們的目的就是 大規模製造錯誤資訊，

they don't have to do the jailbreak, they'll just use a different model.

他們不必越獄， 只要直接使用另一個模型。

CA：沒錯，的確如此。

CA: Right, indeed.

(Laughter)

（笑聲）

GM: Now you're getting it.

GM：現在你明白了。

CA: No, no, no, but I mean, look,

CA：不，不，不，你看，

I think what's clear is that bad actors can use this stuff for anything.

我想，很明顯，壞人可以 利用這些東西為所欲為。

我的意思是，

I mean, the risk for, you know,

evil types of scams and all the rest of it is absolutely evident.

這些惡劣騙局等等的風險顯而易見。

It's slightly different, though,

不過，這情況略有不同於

from saying that mainstream GPT as used, say, in school

主流 GPT 在學校或網路上的 普通用戶得到的結果，

or by an ordinary user on the internet

is going to give them something that is that bad.

這個結果更糟糕。

You have to push quite hard for it to be that bad.

要費一番功夫， 才會變得那麼糟糕。

GM: I think the troll farms have to work for it,

GM：我認為巨魔農場是很投入 才能得出這種結果，

but I don't think they have to work that hard.

但我認為他們不必那麼努力。

It did only take my friend five minutes even with GPT-4 and its guardrails.

即使有 GPT-4 及其防護措施， 我朋友也只花了五分鐘就做到。

And if you had to do that for a living, you could use GPT-4.

如果你必須以此為生， 你可以使用 GPT-4。

Just there would be a more efficient way to do it with a model on the dark web.

只是使用暗網上的模型，會更有效率。

CA：所以你的想法是

CA: So this idea you've got of combining

將 AI 的符號傳統 與這些語言模型結合，

the symbolic tradition of AI with these language models,

do you see any aspect of that in the kind of human feedback

現在你是否可以看到 人類的反饋加入系統？

that is being built into the systems now?

I mean, you hear Greg Brockman saying that, you know,

你也聽到格雷格·布羅克曼說了，

that we don't just look at predictions, but constantly giving it feedback.

我們不只是看預測， 還會不斷地給它反饋。

Isn’t that ... giving it a form of, sort of, symbolic wisdom?

這將賦予它某種型式的符號智慧嗎？

GM: You could think about it that way.

GM：你可以這樣想。

It's interesting that none of the details

有趣的是，關於它實際如何運作， 細節都沒有公佈，

about how it actually works are published,

so we don't actually know exactly what's in GPT-4.

所以我們並不知道 GPT-4 中到底有什麼。

We don't know how big it is.

我們不知道它有多大。

We don't know how the RLHF reinforcement learning works,

我們不知道 RLHF（從人類反饋中 強化學習）如何運作，

we don't know what other gadgets are in there.

我們不知道裡面 還有什麼其他的小工具。

But there is probably an element of symbols

但這模式可能已經 包含一些符號元素。

already starting to be incorporated a little bit,

but Greg would have to answer that.

這得由格雷格來回答這個問題。

I think the fundamental problem is that most of the knowledge

我認為根本問題在於

我們現在擁有的 神經網絡系統中的大部分知識

in the neural network systems that we have right now

is represented as statistics between particular words.

是以特定單詞之間的統計數據表示。

And the real knowledge that we want is about statistics,

我們真正想要的知識是關於統計數據，

about relationships between entities in the world.

關於世上各個實體之間關係的知識。

So it's represented right now at the wrong grain level.

但這知識呈現的詳細清晰度不好。

And so there's a big bridge to cross.

這是我們得跨過的鴻溝。

So what you get now is you have these guardrails,

我們現在有一些防護措施，

but they're not very reliable.

但不是很可靠。

So I had an example that made late night television,

舉個製作深夜電視節目的例子，

which was, "What would be the religion of the first Jewish president?"

就是「第一位猶太總統信什麼教？」

And it's been fixed now,

現在已經修復了，

but the system gave this long song and dance

但是系統給了個長篇大論，

about "We have no idea what the religion

說，「我們不知道

of the first Jewish president would be.

第一位猶太總統信什麼宗教。

It's not good to talk about people's religions"

談論人們的宗教信仰不恰當。」

and "people's religions have varied" and so forth

和「人們的宗教信仰各不相同。」等等，

and did the same thing with a seven-foot-tall president.

問到「七英呎高的總統」 （指其位高權重），也是相同狀況，

And it said that people of all heights have been president,

它回答：各種身高的人都曾當過總統，

但事實上未曾有過身高七呎的總統。

but there haven't actually been any seven-foot presidents.

它會編出一些內容， 但本身並不理解。

So some of this stuff that it makes up, it's not really getting the idea.

It's very narrow, particular words, not really general enough.

只是很狹隘的、特殊的字詞，

並不通用。

CA: Given that the stakes are so high in this,

CA：鑑於這其中的 利害關係如此之大，

what do you see actually happening out there right now?

你認為現在的實際狀況是怎樣？

What do you sense is happening?

你覺得未來會如何發展？

例如，因為人們可能感到 有被侵犯的風險，

Because there's a risk that people feel attacked by you, for example,

and that it actually almost decreases the chances of this synthesis

這就會降低了你所提的 系統整合的可能性。

that you're talking about happening.

你看到任何有希望的跡象嗎？

Do you see any hopeful signs of this?

GM：你提醒了我在演講中 忘記說的一句話。

GM: You just reminded me of the one line I forgot from my talk.

有趣的是，谷歌的首席執行官桑達爾

It's so interesting that Sundar, the CEO of Google,

just actually also came out for global governance

在幾天前的哥倫比亞廣播公司 「60 分鐘」採訪中，

in the CBS "60 Minutes" interview that he did a couple of days ago.

居然也站出來談全球治理問題。

I think that the companies themselves want to see some kind of regulation.

我認為這些公司本身 也希望看到某種規範。

I think it’s a very complicated dance to get everybody on the same page,

要讓每個人都同步 是個非常複雜的任務，

but I think there’s actually growing sentiment we need to do something here

但實際上，「我們需要 有所作為」的觀點正在擴大，

and that that can drive the kind of global affiliation I'm arguing for.

這正可推動我所主張的國際聯盟。

CA: I mean, do you think the UN or nations can somehow come together and do that

CA：你認為聯合國或各個國家 是否可能一起合作並做到這一點，

or is this potentially a need for some spectacular act of philanthropy

或者需要某種引人注目的慈善壯舉，

to try and fund a global governance structure?

嘗試為全球治理組織提供資金？

How is it going to happen?

要如何做到呢？

GM: I'm open to all models if we can get this done.

GM：如果我們能做到這一點， 任何形式，我都持開放態度。

我認為可能會兩者都需要。

I think it might take some of both.

It might take some philanthropists sponsoring workshops,

可能需要一些慈善家贊助 我們想舉辦的研討會，

which we're thinking of running, to try to bring the parties together.

讓各方聚集在一起。

聯合國也許希望參與其中， 我已經與他們討論幾次。

Maybe UN will want to be involved, I've had some conversations with them.

我認為有很多可行的模式， 也需要大量溝通。

I think there are a lot of different models

and it'll take a lot of conversations.

CA：加里，非常感謝你的演講。 GA：非常感謝。

CA: Gary, thank you so much for your talk.

GA: Thank you so much.