How to get better at video games, according to babies - Brian Christian

翻译人员: psjmz mz 校对人员: Helen Chang

In 2013, a group of researchers at DeepMind in London

2013年一群伦敦 DeepMind公司的研究者

had set their sights on a grand challenge.

把目光放在一大挑战上。

They wanted to create an AI system that could beat,

他们想要创建一个人工智能系统， 不仅能胜一个，

not just a single Atari game, but every Atari game.

而是能够全部完胜雅达利 （Atari）游戏。

They developed a system they called Deep Q Networks, or DQN,

他们开发了个名叫 强化学习的网络（DQN），

and less than two years later, it was superhuman.

在不到两年，它超越人类。

DQN was getting scores 13 times better

DQN打砖块游戏（Breakout）的得分

than professional human games testers at “Breakout,”

比人类专业游戏玩家高13倍，

17 times better at “Boxing,” and 25 times better at “Video Pinball.”

在拳击游戏中高17倍， 在电子弹珠台中高25倍。

But there was one notable, and glaring, exception.

但有一很明显的例外。

When playing “Montezuma’s Revenge” DQN couldn’t score a single point,

玩游戏《Montezuma’s Revenge》时， DQN一分都拿不到，

even after playing for weeks.

即便玩了几周后。

What was it that made this particular game so vexingly difficult for AI?

是什么让这个特别的游戏 对人工智能这么难胜？

And what would it take to solve it?

需要采取什么来解决它？

Spoiler alert: babies.

剧透警告：婴儿。

We’ll come back to that in a minute.

我们1分钟后回来。

Playing Atari games with AI involves what’s called reinforcement learning,

人工智能玩雅达利游戏 涉及到强化学习，

where the system is designed to maximize some kind of numerical rewards.

在这里系统被设计为 最大化某种量化的奖励。

In this case, those rewards were simply the game's points.

在这个例子中，这些奖励是游戏分数。

This underlying goal drives the system to learn which buttons to press

这个潜在的目标驱使系统 学习按哪个按键

and when to press them to get the most points.

以及何时去按来获得最高分数。

Some systems use model-based approaches, where they have a model of the environment

一些系统使用基于模型的方法， 它们有一个环境的模型

that they can use to predict what will happen next

这样它们就能用来预测 一旦它们采取特定行动后，

once they take a certain action.

下一步会发生什么。

DQN, however, is model free.

然而，DQN没有任何模型。

Instead of explicitly modeling its environment,

与其明确地建模环境，

it just learns to predict, based on the images on screen,

它只需要基于屏幕上的图像学习预测，

how many future points it can expect to earn by pressing different buttons.

它们按不同的键能够 期望获得多少分数。

For instance, “if the ball is here and I move left, more points,

例如，“如果球在这里， 我向左移就得更多的分数，

but if I move right, no more points.”

但如果向右移就不得分。”

But learning these connections requires a lot of trial and error.

但学习这些联系需要大量的试错。

The DQN system would start by mashing buttons randomly,

DQN系统从随意敲按键开始，

and then slowly piece together which buttons to mash when

然后慢慢拼凑 何时需要敲哪个按键

in order to maximize its score.

才能够得到最高分。

But in playing “Montezuma’s Revenge,”

但在玩《Montezuma’s Revenge》时，

this approach of random button-mashing fell flat on its face.

这种随意敲按键的方法彻底失效了。

A player would have to perform this entire sequence

玩家得做完这整个序列动作

just to score their first points at the very end.

才能最终得到第一分。

A mistake? Game over.

犯个错误？游戏结束。

So how could DQN even know it was on the right track?

那么DQN如何知道它在正确的道路上？

This is where babies come in.

婴儿上场的时候到了。

In studies, infants consistently look longer at pictures

在研究中，婴儿看没见过的图片要比

they haven’t seen before than ones they have.

见过的图片花更多的时间。

There just seems to be something intrinsically rewarding about novelty.

新奇似乎就是某种内在奖励。

This behavior has been essential in understanding the infant mind.

这种行为对于理解 婴儿的心理至关重要。

It also turned out to be the secret to beating “Montezuma’s Revenge.”

这正好也是玩好 《Montezuma’s Revenge》游戏的秘密。

The DeepMind researchers worked out an ingenious way

DeepMind研究人员找到 巧妙的方法

to plug this preference for novelty into reinforcement learning.

将这种对新奇事物的偏好 插入到强化学习中。

They made it so that unusual or new images appearing on the screen

他们让不同寻常 或新的图片出现在屏幕中时

were every bit as rewarding as real in-game points.

与真正的游戏积分一样有奖励意义。

Suddenly, DQN was behaving totally differently from before.

突然之间，DQN的行为 完全跟起初不一样了。

It wanted to explore the room it was in,

它想要探索所处的房间，

to grab the key and escape through the locked door—

去抓住钥匙并 通过锁住的门逃出去——

not because it was worth 100 points,

不是因为这价值100分，

but for the same reason we would: to see what was on the other side.

而是跟我们的理由一样： 去看看另一边有什么。

With this new drive, DQN not only managed to grab that first key—

通过这种新的激励，DQN不仅 能够抓住第一把钥匙——

it explored all the way through 15 of the temple’s 24 chambers.

它还在24个房间中，探索了15个房间。

But emphasizing novelty-based rewards can sometimes create more problems

但强调基于新奇的奖励有时候 会带来比它解决的问题

than it solves.

更多的问题。

A novelty-seeking system that’s played a game too long

一个新颖性探索的系统 如果玩游戏太久

will eventually lose motivation.

最终会失去动力。

If it’s seen it all before, why go anywhere?

如果这都是以前见过的， 为什么还要去呢？

Alternately, if it encounters, say, a television, it will freeze.

换之，假如它遇到电视， 它就会停下来。

The constant novel images are essentially paralyzing.

不断出现的新奇图像基本让人瘫痪。

The ideas and inspiration here go in both directions.

这个想法和启发是双向的。

AI researchers stuck on a practical problem,

人工智能研究人员被一个问题困住了，

like how to get DQN to beat a difficult game,

比如如何让DQN打赢一个不同的游戏，

are turning increasingly to experts in human intelligence for ideas.

逐渐变成了探索人类的思想智能。

At the same time,

同时，

AI is giving us new insights into the ways we get stuck and unstuck:

人工智能给我们提供了新的视角， 让我们了解如何陷入和摆脱困境：

into boredom, depression, and addiction,

变得无聊、沮丧和上瘾，

along with curiosity, creativity, and play.

还有好奇心、创造力和玩乐。