How to get better at video games, according to babies - Brian Christian

譯者: Lilian Chiu 審譯者: Helen Chang

In 2013, a group of researchers at DeepMind in London

2013 年，倫敦的 DeepMind 公司有一群研究員

had set their sights on a grand challenge.

決定要接受一項大挑戰。

They wanted to create an AI system that could beat,

他們要開發一個人工智慧系統，

不只能贏一場雅達利（Atari）遊戲，

not just a single Atari game, but every Atari game.

而是贏所有的雅達利遊戲。

They developed a system they called Deep Q Networks, or DQN,

他們開發了一個系統， 叫做深度 Q 網路（DQN）。

and less than two years later, it was superhuman.

不到兩年，它就超越了人類。

DQN was getting scores 13 times better

DQN 玩《Breakout》的得分

比專業的人類遊戲測試者高十三倍，

than professional human games testers at “Breakout,”

17 times better at “Boxing,” and 25 times better at “Video Pinball.”

玩《Boxing》的得分是十七倍，

《Video Pinball》是二十五倍。

But there was one notable, and glaring, exception.

但有個很引人注意且顯目的例外。

When playing “Montezuma’s Revenge” DQN couldn’t score a single point,

在玩《Montezuma’s Revenge》時，

DQN 一分也得不到，

even after playing for weeks.

玩了幾週之後仍然如此。

What was it that made this particular game so vexingly difficult for AI?

是什麼原因讓這款遊戲 特別讓人工智慧傷腦筋？

And what would it take to solve it?

要靠什麼才能解決這個問題？

Spoiler alert: babies.

以下有雷：

嬰兒。

We’ll come back to that in a minute.

這部分我們等下再回來談。

Playing Atari games with AI involves what’s called reinforcement learning,

用人工智慧玩雅達利 遊戲會用到所謂的

強化學習，

where the system is designed to maximize some kind of numerical rewards.

系統的設計是會讓某種 數字化的獎勵達到最高。

In this case, those rewards were simply the game's points.

在這個例子中， 獎勵就是遊戲的得分。

This underlying goal drives the system to learn which buttons to press

這個背後的目標，會驅使 系統去學習要按哪個按鈕，

and when to press them to get the most points.

以及何時要按， 才能得到最高的分數。

Some systems use model-based approaches, where they have a model of the environment

有些系統用以模型為基礎的方法，

會有一個環境的模型，

that they can use to predict what will happen next

用這個模型來預測 採取某個行動的後果。

once they take a certain action.

DQN, however, is model free.

然而，DQN 是不用模型的。

Instead of explicitly modeling its environment,

它不明確地將環境建模，

it just learns to predict, based on the images on screen,

而是學習根據螢幕上的影像來預測

how many future points it can expect to earn by pressing different buttons.

按各個按鈕預期將會得多少分。

For instance, “if the ball is here and I move left, more points,

比如：「如果球在這裡， 而我向左移動，

就會有更多分數，但若向右， 就沒有更多分數。」

but if I move right, no more points.”

But learning these connections requires a lot of trial and error.

但學習這些連結需要很大量的試誤。

The DQN system would start by mashing buttons randomly,

DQN 系統一開始是 先隨機亂按按鈕，

and then slowly piece together which buttons to mash when

接著慢慢拼湊出何時要按哪些按鈕

in order to maximize its score.

才能讓分數達到最高。

But in playing “Montezuma’s Revenge,”

但在玩《Montezuma’s Revenge》時，

this approach of random button-mashing fell flat on its face.

用隨機按鈕的方法輸得慘兮兮。

A player would have to perform this entire sequence

玩家得要做完這一連串過程

just to score their first points at the very end.

才能在最後得到第一分。

A mistake? Game over.

犯一個錯呢？遊戲結束。

So how could DQN even know it was on the right track?

所以 DQN 怎麼會知道 它走在對的路徑上了？

This is where babies come in.

此時就需要嬰兒了。

In studies, infants consistently look longer at pictures

在研究中，比起見過的圖片，嬰兒會

很一致地花更多時間 去看他們以前沒見過的圖片。

they haven’t seen before than ones they have.

There just seems to be something intrinsically rewarding about novelty.

新奇感在本質上似乎很有回饋價值。

This behavior has been essential in understanding the infant mind.

對於了解嬰兒的大腦， 這種行為十分重要。

It also turned out to be the secret to beating “Montezuma’s Revenge.”

後來也發現它就是贏得 《Montezuma’s Revenge》的秘密。

The DeepMind researchers worked out an ingenious way

DeepMind 研究者想出了 一種很別出心裁的方法，

to plug this preference for novelty into reinforcement learning.

把這種對新奇性的偏好

放入到強化學習中。

They made it so that unusual or new images appearing on the screen

他們讓不尋常或新的 影像出現在螢幕時

were every bit as rewarding as real in-game points.

就和遊戲中的得分一樣有價值。

Suddenly, DQN was behaving totally differently from before.

突然間，DQN 的行為 和以前完全不同了。

It wanted to explore the room it was in,

它會想要探索它所處的房間，

to grab the key and escape through the locked door—

抓起鑰匙，從鎖住的門逃脫——

not because it was worth 100 points,

理由不是因為這樣做能得一百分，

but for the same reason we would: to see what was on the other side.

而是和我們一樣的理由：

想看看另一頭有什麼。

With this new drive, DQN not only managed to grab that first key—

有了這個新動機，

DQN 不僅想辦法取得了 第一把鑰匙——

it explored all the way through 15 of the temple’s 24 chambers.

它還一路探索了廟裡 二十四個房間當中的十五個。

But emphasizing novelty-based rewards can sometimes create more problems

但著重以新奇性為基礎的獎勵， 有時製造的問題比解決的還多。

than it solves.

A novelty-seeking system that’s played a game too long

尋求新奇性的系統 如果玩一個遊戲太久，

will eventually lose motivation.

終究會失去動力。

If it’s seen it all before, why go anywhere?

如果什麼都看過了， 那何必還要去任何地方？

Alternately, if it encounters, say, a television, it will freeze.

另一方面，比如， 如果它遇到了一台電視，

它會呆住。

The constant novel images are essentially paralyzing.

不斷出現的新奇影像 最終會讓它癱瘓。

The ideas and inspiration here go in both directions.

這裡的想法和鼓舞都是雙向的。

AI researchers stuck on a practical problem,

人工智慧研究者若卡在 實際的問題上，比如

like how to get DQN to beat a difficult game,

如何在困難的遊戲中讓 DQN 能贏，

are turning increasingly to experts in human intelligence for ideas.

他們越來越向人類智慧的 專家尋求點子。

At the same time,

同時，

AI is giving us new insights into the ways we get stuck and unstuck:

人工智慧也讓我們對於我們卡住 和脫身的方式有了新的洞見：

into boredom, depression, and addiction,

思考無聊、沮喪，和成癮，

along with curiosity, creativity, and play.

同時也思考好奇心、創意，和玩樂。