What we learned from 5 million books

翻译人员: Lili Liang 校对人员: dahong zhang

Erez Lieberman Aiden: Everyone knows

Erez Liberman Aiden：人说

that a picture is worth a thousand words.

一副画面抵过一千个词

But we at Harvard

但是我们在哈佛大学

were wondering if this was really true.

却在思考这是不是一定正确

(Laughter)

（众人笑）

So we assembled a team of experts,

我们召集了各方专家

spanning Harvard, MIT,

他们来自哈佛 麻省理工

The American Heritage Dictionary, The Encyclopedia Britannica

《英国大百科全书》 《美国传统英语字典》

and even our proud sponsors,

还有我们骄傲的赞助商

the Google.

谷歌

And we cogitated about this

我们思考了

for about four years.

大概四年

And we came to a startling conclusion.

最后得出一个惊人的结论

Ladies and gentlemen, a picture is not worth a thousand words.

女士们先生们 一副画面可不止一千个词那么简单

In fact, we found some pictures

事实上 我们发现有时候

that are worth 500 billion words.

一幅画面抵过5千亿个词

Jean-Baptiste Michel: So how did we get to this conclusion?

Jean-Baptiste Michel: 我们是如何得出这个结论的呢

So Erez and I were thinking about ways

是这样的 Erez和我

to get a big picture of human culture

在想怎样找到一幅展现人类文明

and human history: change over time.

和人文历史的画面： 历史的变迁

So many books actually have been written over the years.

人们在漫长岁月中写了很多书

So we were thinking, well the best way to learn from them

所以我们想 向他们学习的最佳方法

is to read all of these millions of books.

就是把那几百万本书全都读完

Now of course, if there's a scale for how awesome that is,

当然 如果用坐标来表示这样做的好处

that has to rank extremely, extremely high.

那Y轴上的值一定是极高的

Now the problem is there's an X-axis for that,

但问题是还有X轴

which is the practical axis.

也就是可行性

This is very, very low.

这是极低的

(Applause)

（众人鼓掌）

Now people tend to use an alternative approach,

现在人们倾向于另一种做法

which is to take a few sources and read them very carefully.

那就是选择几本书进行精读

This is extremely practical, but not so awesome.

可行性极高但还不够好

What you really want to do

人们真正想要的

is to get to the awesome yet practical part of this space.

是一个既好又可行的方法

So it turns out there was a company across the river called Google

结果 在水一方 有一家叫“谷歌”的公司

who had started a digitization project a few years back

他们在此之前的几年前就开始了一个数字化工程

that might just enable this approach.

有可能帮我们找到这个“既好又可行”的方法

They have digitized millions of books.

他们已经将几百万本书进行了数字化

So what that means is, one could use computational methods

这就意味着人们在电脑上点几个键

to read all of the books in a click of a button.

就能阅读所有的书

That's very practical and extremely awesome.

这真的是既可行又好

ELA: Let me tell you a little bit about where books come from.

这些书是哪里来的呢

Since time immemorial, there have been authors.

从古时候开始 人们就开始写作了

These authors have been striving to write books.

这些作家写书都非常卖力

And this became considerably easier

几个世纪前印刷机问世了

with the development of the printing press some centuries ago.

写书的过程变得简单多了

Since then, the authors have won

自那以后

on 129 million distinct occasions,

作家们已经出版了

publishing books.

1.29亿本书

Now if those books are not lost to history,

如果这些书没有随年月而遗失

then they are somewhere in a library,

就都在图书馆里存着

and many of those books have been getting retrieved from the libraries

谷歌已经把许多书从图书馆中调了出来

and digitized by Google,

进行了数字化

which has scanned 15 million books to date.

被扫描的书籍到目前已有1500万册

Now when Google digitizes a book, they put it into a really nice format.

谷歌扫描图书时 把书的格式做得很好

Now we've got the data, plus we have metadata.

现在我们不但有了数据 还有元数据

We have information about things like where was it published,

我们掌握了这些书的出版地

who was the author, when was it published.

作者 出版时间等信息

And what we do is go through all of those records

接下来 我们就要从所有这些记录中

and exclude everything that's not the highest quality data.

筛选出质量最高的数据

What we're left with

最后剩下的

is a collection of five million books,

是5百万本书

500 billion words,

5000亿个词

a string of characters a thousand times longer

这么多词连起来

than the human genome --

长度是人类基因组的1000倍

a text which, when written out,

如果把这些词连续写出来

would stretch from here to the Moon and back

其长度相当于在地月之间

10 times over --

往返10次以上

a veritable shard of our cultural genome.

这还仅是我们文化基因组的小小一段

Of course what we did

当然啦

when faced with such outrageous hyperbole ...

面对如此令人崩溃的结果

(Laughter)

（众人笑）

was what any self-respecting researchers

我们做了一个懂得自重的研究者

would have done.

应该做的事

We took a page out of XKCD,

我们借鉴了XKCD（科学漫画）

and we said, "Stand back.

说：" 往后站。

We're going to try science."

我们要用科学来解决问题。”

(Laughter)

（众人笑）

JM: Now of course, we were thinking,

当然 这时我们在想

well let's just first put the data out there

何不先把数据放上去

for people to do science to it.

让人们通过科学来运用数据

Now we're thinking, what data can we release?

现在我们在思考 哪些数据可以公开

Well of course, you want to take the books

你当然想把这所有5百万本书

and release the full text of these five million books.

全文公开

Now Google, and Jon Orwant in particular,

现在谷歌 具体地说是乔恩. 奥温特

told us a little equation that we should learn.

告诉教给我们一个有用的方程式

So you have five million, that is, five million authors

你有5百万本书 那就有五百万个作者

and five million plaintiffs is a massive lawsuit.

一个有5百万个原告的官司可不小啊

So, although that would be really, really awesome,

所以尽管这是个好想法

again, that's extremely, extremely impractical.

但是也极不现实

(Laughter)

（众人笑）

Now again, we kind of caved in,

现在我们做出些许让步

and we did the very practical approach, which was a bit less awesome.

采用一个非常可行但稍微没那么好的方法

We said, well instead of releasing the full text,

我们不公开全书内容

we're going to release statistics about the books.

而是公开书本的相关统计数据

So take for instance "A gleam of happiness."

拿“A gleam of happiness”这个词组做例子

It's four words; we call that a four-gram.

它有四个单词 我们称它为四字格

We're going to tell you how many times a particular four-gram

我们会告诉你直到2008年出版的书中

appeared in books in 1801, 1802, 1803,

在1801年 1802年 1803年一直到2008年

all the way up to 2008.

某个四字格一共出现了多少次

That gives us a time series

这让我们看到

of how frequently this particular sentence was used over time.

这个词组在这段时期内被使用的频率

We do that for all the words and phrases that appear in those books,

我们对在这些书中的所有单词和词组都这么处理

and that gives us a big table of two billion lines

于是我们得出了一个由20亿曲线

that tell us about the way culture has been changing.

表示出文化变化的情况

ELA: So those two billion lines,

这20亿条曲线

we call them two billion n-grams.

我们成作20亿个n字格

What do they tell us?

它们告诉了我们什么

Well the individual n-grams measure cultural trends.

这些n字格衡量的是文化的走势

Let me give you an example.

我来举个例子

Let's suppose that I am thriving,

假设 我正在发财

then tomorrow I want to tell you about how well I did.

明天我告诉你我发财的情况

And so I might say, "Yesterday, I throve."

我会说：“昨天，我发了。”

Alternatively, I could say, "Yesterday, I thrived."

也可以说：“昨天，我发财了。”

Well which one should I use?

我到底应该用哪个说法呢

How to know?

怎么找答案

As of about six months ago,

6个月以前

the state of the art in this field

很流行的做法是

is that you would, for instance,

比如说

go up to the following psychologist with fabulous hair,

你去问这位秀发飘逸的心理学家

and you'd say,

你说

"Steve, you're an expert on the irregular verbs.

“史蒂夫，你是不规则动词的专家。

What should I do?"

我该怎么办啊？”

And he'd tell you, "Well most people say thrived,

他会说：“大多数人说‘发财了’，

but some people say throve."

但有些人说‘发了’。”

And you also knew, more or less,

如果你可以

that if you were to go back in time 200 years

回到200年前

and ask the following statesman with equally fabulous hair,

问问这位秀发同样飘逸的政治家

(Laughter)

（众人笑）

"Tom, what should I say?"

“托马斯，我该怎么说？”

He'd say, "Well, in my day, most people throve,

他会回答：“嗯，在我的时代，大多数人说‘发了’，

but some thrived."

但是少数人说‘发财了’。”

So now what I'm just going to show you is raw data.

现在我给你们看一个原始数据

Two rows from this table of two billion entries.

这是20亿本书中的其中两本书的曲线

What you're seeing is year by year frequency

你们将看到“发了”和“发财了”这两个词

of "thrived" and "throve" over time.

随时间的推移被使用的频率

Now this is just two

这还只是

out of two billion rows.

20亿条曲线中的其中两条

So the entire data set

整套数据

is a billion times more awesome than this slide.

比这张幻灯片要宏伟10亿倍

(Laughter)

（众人笑）

(Applause)

（众人鼓掌）

JM: Now there are many other pictures that are worth 500 billion words.

很多画面都相当于5千亿个词

For instance, this one.

比如这一幅

If you just take influenza,

如果你找“流行感冒”这一词

you will see peaks at the time where you knew

你会看到几个全球范围内

big flu epidemics were killing people around the globe.

祸害人命的流感高峰

ELA: If you were not yet convinced,

如果这不足以令人信服

sea levels are rising,

海平面正在上升

so is atmospheric CO2 and global temperature.

大气中二氧化碳含量和全球气温都在升高

JM: You might also want to have a look at this particular n-gram,

你们也可以看看这个n字格

and that's to tell Nietzsche that God is not dead,

告诉尼采上帝没死

although you might agree that he might need a better publicist.

你可能也认为他或许要换一个企宣了

(Laughter)

（众人笑）

ELA: You can get at some pretty abstract concepts with this sort of thing.

你可以通过这个得到非常抽象的概念

For instance, let me tell you the history

我跟你们说说

of the year 1950.

1950年的历史

Pretty much for the vast majority of history,

在漫漫历史长河中

no one gave a damn about 1950.

几乎没人在意1950年

In 1700, in 1800, in 1900,

1700年 1800年 1900年

no one cared.

没有人在意

Through the 30s and 40s,

20世纪三十年代和四十年代

no one cared.

没有人在意

Suddenly, in the mid-40s,

到了四十年代中期 突然间

there started to be a buzz.

关注度飞升

People realized that 1950 was going to happen,

人们意识到1950年快来了

and it could be big.

这一年可能非同小可啊

(Laughter)

（众人笑）

But nothing got people interested in 1950

1950年 正如人们想象的一样

like the year 1950.

没发生任何有意思的事情

(Laughter)

（众人笑）

People were walking around obsessed.

人们都着了魔了

They couldn't stop talking

无时无刻不在谈论

about all the things they did in 1950,

他们1950年做过的事情

all the things they were planning to do in 1950,

他们打算在1950年做的事情

all the dreams of what they wanted to accomplish in 1950.

后者他们1950年想要实现的梦想

In fact, 1950 was so fascinating

事实上 1950年是不同凡响的一年

that for years thereafter,

即使过了好多年

people just kept talking about all the amazing things that happened,

人们还是不停地谈论那年发生的所有美好事情

in '51, '52, '53.

51年 52年 53年

Finally in 1954,

终于到了1954年

someone woke up and realized

人们醒悟过来

that 1950 had gotten somewhat passé.

1950年已成往事了

(Laughter)

（众人笑）

And just like that, the bubble burst.

就这样 泡泡破了

(Laughter)

（众人笑）

And the story of 1950

1950年的情况

is the story of every year that we have on record,

以及每一年的情况 我们都记录了下来

with a little twist, because now we've got these nice charts.

多亏了这些漂亮的图表 我们的工作顺利多了

And because we have these nice charts, we can measure things.

有了这些漂亮的图表 我们就能测量各种事物

We can say, "Well how fast does the bubble burst?"

我们会说：“泡泡破掉的速度有多快？”

And it turns out that we can measure that very precisely.

结果证明 我们可以对此进行精准的测量

Equations were derived, graphs were produced,

等式出来了 图表也做好了

and the net result

最终结果是

is that we find that the bubble bursts faster and faster

泡泡破掉的速度

with each passing year.

每年都在加快

We are losing interest in the past more rapidly.

我们对过去的遗忘不断加快

JM: Now a little piece of career advice.

好 现在给大家一些发展事业的建议

So for those of you who seek to be famous,

如果你想成名

we can learn from the 25 most famous political figures,

我们可以向25位最著名的政治人物

authors, actors and so on.

作家 演员学习

So if you want to become famous early on, you should be an actor,

如果你想早点成名 你就应该做个演员

because then fame starts rising by the end of your 20s --

因为 演员在20来岁的时候成名

you're still young, it's really great.

你还很年轻 这是本钱

Now if you can wait a little bit, you should be an author,

如果你能等一等 那就当个作家

because then you rise to very great heights,

因为你可以像马克.吐温这样

like Mark Twain, for instance: extremely famous.

成为文坛巨星

But if you want to reach the very top,

如果你想到达万人之上

you should delay gratification

你就不能安于现状

and, of course, become a politician.

要成为一个政治家

So here you will become famous by the end of your 50s,

到了快60岁的时候 你就成名了

and become very, very famous afterward.

而且之后名声远扬

So scientists also tend to get famous when they're much older.

科学家通常在年纪一大把的时候才成名

Like for instance, biologists and physics

生物学家和物理学家的名声

tend to be almost as famous as actors.

通常能跟演员的名声媲美

One mistake you should not do is become a mathematician.

有一个错误你不要犯 那就是成为一个数学家

(Laughter)

（众人笑）

If you do that,

如果你成了数学家

you might think, "Oh great. I'm going to do my best work when I'm in my 20s."

你会想：“太好啦，我20多岁的时候会有最辉煌的成就。”

But guess what, nobody will really care.

谁知道 人们连睬都不睬你

(Laughter)

（众人笑）

ELA: There are more sobering notes

n字格中

among the n-grams.

有些情况更为明了

For instance, here's the trajectory of Marc Chagall,

这是Marc Chagall的名声起落

an artist born in 1887.

他是出生于1887的一位艺术家

And this looks like the normal trajectory of a famous person.

他的名声起落看似乎没有什么异常

He gets more and more and more famous,

他的名声越来越大

except if you look in German.

然而如果你在德语书中搜索 情况就不同了

If you look in German, you see something completely bizarre,

在德语书中 你会看到非常奇怪的现象

something you pretty much never see,

闻所未闻 见所未见

which is he becomes extremely famous

他先是名极一时

and then all of a sudden plummets,

但突然之间 名声直线下落

going through a nadir between 1933 and 1945,

在1933年到1945年间达到了低谷

before rebounding afterward.

后来才回升

And of course, what we're seeing

当然 实际情况是

is the fact Marc Chagall was a Jewish artist

Marc Chagall是一个犹太艺术家

in Nazi Germany.

当时身在纳粹德国

Now these signals

这些信号

are actually so strong

实在太强了

that we don't need to know that someone was censored.

我们无需知道谁被禁了

We can actually figure it out

我们事实上可以

using really basic signal processing.

通过非常基本的信号处理来找出答案

Here's a simple way to do it.

这里有一个简单的方法

Well, a reasonable expectation

一个人在特定时期内

is that somebody's fame in a given period of time

所拥有的知名度

should be roughly the average of their fame before

应当大致为他成名前与成名后知名度的平均值

and their fame after.

这么想是有道理的

So that's sort of what we expect.

我们也是怎么想的

And we compare that to the fame that we observe.

我们把观察到的知名度进行对比

And we just divide one by the other

我们把前者比上后者

to produce something we call a suppression index.

产生的结果叫做抑制指数

If the suppression index is very, very, very small,

如果抑制指数非常非常小

then you very well might be being suppressed.

那么你的知名度正在被抑制

If it's very large, maybe you're benefiting from propaganda.

如果数值非常大 或许就表明你从宣传中获益

JM: Now you can actually look at

你还可以看到

the distribution of suppression indexes over whole populations.

压抑指数在总人数中的分布情况

So for instance, here --

这里有个例子

this suppression index is for 5,000 people

这是从没有明显抑制的英文书籍中

picked in English books where there's no known suppression --

选出的5000个人

it would be like this, basically tightly centered on one.

它是这个样子的 基本上以1为中心

What you expect is basically what you observe.

实际情况与预想差不多

This is distribution as seen in Germany --

而这在是德文书籍中的分布情况

very different, it's shifted to the left.

与前者大为不同 往左偏了

People talked about it twice less as it should have been.

人们对它的关注较预期要少了两倍

But much more importantly, the distribution is much wider.

更重要的是 这个分布的跨度更宽

There are many people who end up on the far left on this distribution

不少人处于左边的部分

who are talked about 10 times fewer than they should have been.

人数比预期中少了10倍

But then also many people on the far right

而也有不少人处于更靠右的部分

who seem to benefit from propaganda.

他们的宣传起了作用

This picture is the hallmark of censorship in the book record.

这幅图反映了书籍记录中的审查情况

ELA: So culturomics

我们把这种方法

is what we call this method.

称作文化组学

It's kind of like genomics.

有点像基因组学

Except genomics is a lens on biology

只不过 基因组学是生物学上

through the window of the sequence of bases in the human genome.

观察人类基因组序列的透镜

Culturomics is similar.

文化组学很类似

It's the application of massive-scale data collection analysis

它指的是对人类文明研究的

to the study of human culture.

大规模数据收集分析的应用

Here, instead of through the lens of a genome,

它使用的不是基因组这个透镜

through the lens of digitized pieces of the historical record.

而是用数字化的历史记录片段作为透镜

The great thing about culturomics

文化组学的优点是

is that everyone can do it.

人人都会用它

Why can everyone do it?

为什么呢

Everyone can do it because three guys,

这是因为这三个人

Jon Orwant, Matt Gray and Will Brockman over at Google,

谷歌的乔恩.奥温特 迈特.格雷和威尔.布洛克曼

saw the prototype of the Ngram Viewer,

看到了n字格后

and they said, "This is so fun.

说：“这太有意思了，

We have to make this available for people."

我们得让所有人都用上它。”

So in two weeks flat -- the two weeks before our paper came out --

于是在我们的论文发表之前的整整两个星期中

they coded up a version of the Ngram Viewer for the general public.

他们编了一个面向公众的Ngram Viewer版本

And so you too can type in any word or phrase that you're interested in

现在你们也可以输入任何你感兴趣的单词或词组

and see its n-gram immediately --

查看它的n字格

also browse examples of all the various books

并阅览所有书籍中

in which your n-gram appears.

出现n字格的例句

JM: Now this was used over a million times on the first day,

这个词在第一天就被使用了超过一百万次

and this is really the best of all the queries.

这真的是最棒的一个搜索词

So people want to be their best, put their best foot forward.

人们总想做到最好 总想展示最好的一面

But it turns out in the 18th century, people didn't really care about that at all.

但是在18世纪 人们对此并不在乎

They didn't want to be their best, they wanted to be their beft.

他们不想做到最好（“best”）而是“beft”

So what happened is, of course, this is just a mistake.

实际上 这是个错别字

It's not that strove for mediocrity,

这并不是因为人们不识字

it's just that the S used to be written differently, kind of like an F.

而是因为当时英文字母S的写法跟现在不同 看起来像F

Now of course, Google didn't pick this up at the time,

当然 谷歌没有意识到这一点

so we reported this in the science article that we wrote.

于是我们对此在论文中做了报告

But it turns out this is just a reminder

这实际上只是一个小提示

that, although this is a lot of fun,

尽管这很有趣

when you interpret these graphs, you have to be very careful,

但是你在解读这些图表时 仍须非常谨慎

and you have to adopt the base standards in the sciences.

你必须遵循基本的科学准则

ELA: People have been using this for all kinds of fun purposes.

人们使用它来寻求各种乐趣

(Laughter)

（众人笑）

Actually, we're not going to have to talk,

我们不打算多说

we're just going to show you all the slides and remain silent.

光给你们看这些幻灯片

This person was interested in the history of frustration.

这个用户对人们烦躁的历史很感兴趣

There's various types of frustration.

这里有不同类型的烦躁

If you stub your toe, that's a one A "argh."

如果你的脚趾被碰了 你会说“啊” （“argh”）

If the planet Earth is annihilated by the Vogons

如果地球被外星人毁灭了

to make room for an interstellar bypass,

开了一条星际航道

that's an eight A "aaaaaaaargh."

那就是“啊啊啊啊啊啊啊啊” （"aaaaaaaargh"）

This person studies all the "arghs,"

这个人研究了不同长短的“啊” （“argh”）

from one through eight A's.

从1个啊到8个啊

And it turns out

结果

that the less-frequent "arghs"

那些使用频率较低的啊

are, of course, the ones that correspond to things that are more frustrating --

代表程度更高的烦躁

except, oddly, in the early 80s.

八十年代是个例外

We think that might have something to do with Reagan.

我们猜这可能跟里根总统有关

(Laughter)

（众人笑）

JM: There are many usages of this data,

这个数据库的用处很多

but the bottom line is that the historical record is being digitized.

但最重要的是这是一个数字化的历史记录

Google has started to digitize 15 million books.

谷歌已经开始对1500万本书进行数字化处理

That's 12 percent of all the books that have ever been published.

其中12%的书已被出版

It's a sizable chunk of human culture.

这是人类文明相当大的一部分

There's much more in culture: there's manuscripts, there newspapers,

而文明还包括更多的内容 有手稿 报纸

there's things that are not text, like art and paintings.

非文字的内容 例如艺术与绘画

These all happen to be on our computers,

这些内容都会出现在我们的电脑上

on computers across the world.

在世界各地的电脑上

And when that happens, that will transform the way we have

如果这成真了

to understand our past, our present and human culture.

我们对过去现在以及人类文明的认识就被改变了

Thank you very much.

非常感谢大家

(Applause)

（众人鼓掌）