The era of blind faith in big data must end | Cathy O'Neil

翻译人员: Lin Zhang 校对人员: Yolanda Zhang

Algorithms are everywhere.

算法无处不在。

They sort and separate the winners from the losers.

他们把成功者和失败者区分开来。

The winners get the job

成功者得到工作

or a good credit card offer.

或是一个很好的信用卡优惠计划。

The losers don't even get an interview

失败者甚至连面试机会都没有，

or they pay more for insurance.

或者要为保险付更多的钱。

We're being scored with secret formulas that we don't understand

我们被不理解的秘密公式打分，

that often don't have systems of appeal.

却并没有上诉的渠道。

That begs the question:

这引出了一个问题：

What if the algorithms are wrong?

如果算法是错误的怎么办？

To build an algorithm you need two things:

构建一个算法需要两个要素：

you need data, what happened in the past,

需要数据，如过去发生的事情，

and a definition of success,

和成功的定义，

the thing you're looking for and often hoping for.

你正在寻找的，通常希望得到的东西。

You train an algorithm by looking, figuring out.

你可以通过观察，理解来训练算法。

The algorithm figures out what is associated with success.

这种算法能找出与成功相关的因素。

What situation leads to success?

什么情况意味着成功？

Actually, everyone uses algorithms.

其实，每个人都使用算法。

They just don't formalize them in written code.

他们只是没有把它们写成书面代码。

Let me give you an example.

举个例子。

I use an algorithm every day to make a meal for my family.

我每天都用一种算法来 为我的家人做饭。

The data I use

我使用的数据

is the ingredients in my kitchen,

就是我厨房里的原料，

the time I have,

我拥有的时间，

the ambition I have,

我的热情，

and I curate that data.

然后我整理了这些数据。

I don't count those little packages of ramen noodles as food.

我不把那种小包拉面算作食物。

(Laughter)

（笑声）

My definition of success is:

我对成功的定义是：

a meal is successful if my kids eat vegetables.

如果我的孩子们肯吃蔬菜， 这顿饭就是成功的。

It's very different from if my youngest son were in charge.

这和我最小的儿子 负责做饭时的情况有所不同。

He'd say success is if he gets to eat lots of Nutella.

他说，如果他能吃很多 Nutella巧克力榛子酱就是成功。

But I get to choose success.

但我可以选择成功。

I am in charge. My opinion matters.

我负责。我的意见就很重要。

That's the first rule of algorithms.

这就是算法的第一个规则。

Algorithms are opinions embedded in code.

算法是嵌入在代码中的观点。

It's really different from what you think most people think of algorithms.

这和你认为大多数人对 算法的看法是不同的。

They think algorithms are objective and true and scientific.

他们认为算法是客观、真实和科学的。

That's a marketing trick.

那是一种营销技巧。

It's also a marketing trick

这也是一种用算法来

to intimidate you with algorithms,

恐吓你的营销手段，

to make you trust and fear algorithms

为了让你信任和恐惧算法

because you trust and fear mathematics.

因为你信任并害怕数学。

A lot can go wrong when we put blind faith in big data.

当我们盲目信任大数据时， 很多人都可能犯错。

This is Kiri Soares. She's a high school principal in Brooklyn.

这是凯丽·索尔斯。 她是布鲁克林的一名高中校长。

In 2011, she told me her teachers were being scored

2011年，她告诉我， 她学校的老师们正在被一个复杂

with a complex, secret algorithm

并且隐秘的算法进行打分，

called the "value-added model."

这个算法被称为“增值模型"。

I told her, "Well, figure out what the formula is, show it to me.

我告诉她，“先弄清楚这个 公式是什么，然后给我看看。

我来给你解释一下。”

I'm going to explain it to you."

她说，“我寻求过这个公式，

She said, "Well, I tried to get the formula,

但是教育部的负责人告诉我这是数学，

but my Department of Education contact told me it was math

and I wouldn't understand it."

给我我也看不懂。”

It gets worse.

更糟的还在后面。

The New York Post filed a Freedom of Information Act request,

纽约邮报提出了“信息自由法”的要求，

来得到所有老师的名字与他们的分数，

got all the teachers' names and all their scores

and they published them as an act of teacher-shaming.

并且他们以羞辱教师的方式 发表了这些数据。

When I tried to get the formulas, the source code, through the same means,

当我试图用同样的方法来获取公式， 源代码的时候，

I was told I couldn't.

我被告知我没有权力这么做。

I was denied.

我被拒绝了。

I later found out

后来我发现，

that nobody in New York City had access to that formula.

纽约市压根儿没有人能接触到这个公式。

No one understood it.

没有人能看懂。

Then someone really smart got involved, Gary Rubinstein.

然后，一个非常聪明的人参与了， 加里·鲁宾斯坦。

He found 665 teachers from that New York Post data

他从纽约邮报的数据中 找到了665名教师，

that actually had two scores.

实际上他们有两个分数。

That could happen if they were teaching

如果他们同时教七年级与八年级的数学，

seventh grade math and eighth grade math.

就会得到两个评分。

He decided to plot them.

他决定把这些数据绘成图表。

Each dot represents a teacher.

每个点代表一个教师。

(Laughter)

（笑声）

What is that?

那是什么？

(Laughter)

（笑声）

That should never have been used for individual assessment.

它永远不应该被用于个人评估。

It's almost a random number generator.

它几乎是一个随机数生成器。

(Applause)

（掌声）

But it was.

但它确实被使用了。

This is Sarah Wysocki.

这是莎拉·维索斯基。

She got fired, along with 205 other teachers,

她连同另外205名教师被解雇了，

from the Washington, DC school district,

都是来自华盛顿特区的学区，

even though she had great recommendations from her principal

尽管她的校长还有学生的

and the parents of her kids.

父母都非常推荐她。

I know what a lot of you guys are thinking,

我知道你们很多人在想什么，

尤其是这里的数据科学家， 人工智能专家。

especially the data scientists, the AI experts here.

You're thinking, "Well, I would never make an algorithm that inconsistent."

你在想，“我可永远不会做出 这样前后矛盾的算法。”

But algorithms can go wrong,

但是算法可能会出错，

even have deeply destructive effects with good intentions.

即使有良好的意图， 也会产生毁灭性的影响。

And whereas an airplane that's designed badly

每个人都能看到一架设计的

crashes to the earth and everyone sees it,

很糟糕的飞机会坠毁在地，

an algorithm designed badly

而一个设计糟糕的算法

can go on for a long time, silently wreaking havoc.

可以持续很长一段时间， 并无声地造成破坏。

This is Roger Ailes.

这是罗杰·艾尔斯。

(Laughter)

（笑声）

He founded Fox News in 1996.

他在1996年创办了福克斯新闻。

More than 20 women complained about sexual harassment.

公司有超过20多名女性曾抱怨过性骚扰。

They said they weren't allowed to succeed at Fox News.

她们说她们不被允许在 福克斯新闻有所成就。

He was ousted last year, but we've seen recently

他去年被赶下台，但我们最近看到

that the problems have persisted.

问题依然存在。

That begs the question:

这引出了一个问题：

What should Fox News do to turn over another leaf?

福克斯新闻应该做些什么改变？

Well, what if they replaced their hiring process

如果他们用机器学习算法

with a machine-learning algorithm?

取代传统的招聘流程呢？

That sounds good, right?

听起来不错，对吧？

Think about it.

想想看。

The data, what would the data be?

数据，这些数据到底是什么？

A reasonable choice would be the last 21 years of applications to Fox News.

福克斯新闻在过去21年的申请函 是一个合理的选择。

Reasonable.

很合理。

What about the definition of success?

那么成功的定义呢？

Reasonable choice would be,

合理的选择将是，

well, who is successful at Fox News?

谁在福克斯新闻取得了成功？

I guess someone who, say, stayed there for four years

我猜的是，比如在那里呆了四年，

and was promoted at least once.

至少得到过一次晋升的人。

Sounds reasonable.

听起来很合理。

And then the algorithm would be trained.

然后这个算法将会被训练。

It would be trained to look for people to learn what led to success,

它会被训练去向人们 学习是什么造就了成功，

what kind of applications historically led to success

什么样的申请函在过去拥有

by that definition.

这种成功的定义。

Now think about what would happen

现在想想如果我们把它

if we applied that to a current pool of applicants.

应用到目前的申请者中会发生什么。

It would filter out women

它会过滤掉女性，

because they do not look like people who were successful in the past.

因为她们看起来不像 在过去取得成功的人。

Algorithms don't make things fair

算法不会让事情变得公平，

如果你只是轻率地， 盲目地应用算法。

if you just blithely, blindly apply algorithms.

They don't make things fair.

它们不会让事情变得公平。

They repeat our past practices,

它们只是重复我们过去的做法，

our patterns.

我们的规律。

They automate the status quo.

它们使现状自动化。

That would be great if we had a perfect world,

如果我们有一个 完美的世界那就太好了，

but we don't.

但是我们没有。

我还要补充一点， 大多数公司都没有令人尴尬的诉讼，

And I'll add that most companies don't have embarrassing lawsuits,

but the data scientists in those companies

但是这些公司的数据科学家

are told to follow the data,

被告知要跟随数据，

to focus on accuracy.

关注它的准确性。

Think about what that means.

想想这意味着什么。

Because we all have bias, it means they could be codifying sexism

因为我们都有偏见， 这意味着他们可以编纂性别歧视

or any other kind of bigotry.

或者任何其他的偏见。

Thought experiment,

思维实验，

because I like them:

因为我喜欢它们：

an entirely segregated society --

一个完全隔离的社会——

racially segregated, all towns, all neighborhoods

种族隔离存在于所有的城镇， 所有的社区，

and where we send the police only to the minority neighborhoods

我们把警察只送到少数族裔的社区

to look for crime.

去寻找犯罪。

The arrest data would be very biased.

逮捕数据将会是十分有偏见的。

What if, on top of that, we found the data scientists

除此之外，我们还会寻找数据科学家

and paid the data scientists to predict where the next crime would occur?

并付钱给他们来预测 下一起犯罪会发生在哪里？

Minority neighborhood.

少数族裔的社区。

Or to predict who the next criminal would be?

或者预测下一个罪犯会是谁？

A minority.

少数族裔。

The data scientists would brag about how great and how accurate

这些数据科学家们 会吹嘘他们的模型有多好，

their model would be,

多精确，

and they'd be right.

当然他们是对的。

Now, reality isn't that drastic, but we do have severe segregations

不过现实并没有那么极端， 但我们确实在许多城市里

in many cities and towns,

有严重的种族隔离，

and we have plenty of evidence

并且我们有大量的证据表明

of biased policing and justice system data.

警察和司法系统的数据存有偏见。

And we actually do predict hotspots,

而且我们确实预测过热点，

places where crimes will occur.

那些犯罪会发生的地方。

And we do predict, in fact, the individual criminality,

我们确实会预测个人犯罪，

the criminality of individuals.

个人的犯罪行为。

The news organization ProPublica recently looked into

新闻机构“人民 (ProPublica)”最近调查了，

one of those "recidivism risk" algorithms,

其中一个称为

as they're called,

“累犯风险”的算法。

并在佛罗里达州的 宣判期间被法官采用。

being used in Florida during sentencing by judges.

Bernard, on the left, the black man, was scored a 10 out of 10.

伯纳德，左边的那个黑人， 10分中得了满分。

Dylan, on the right, 3 out of 10.

在右边的迪伦， 10分中得了3分。

10分代表高风险。 3分代表低风险。

10 out of 10, high risk. 3 out of 10, low risk.

They were both brought in for drug possession.

他们都因为持有毒品 而被带进了监狱。

They both had records,

他们都有犯罪记录，

but Dylan had a felony

但是迪伦有一个重罪

but Bernard didn't.

但伯纳德没有。

This matters, because the higher score you are,

这很重要，因为你的分数越高，

the more likely you're being given a longer sentence.

你被判长期服刑的可能性就越大。

What's going on?

到底发生了什么？

Data laundering.

数据洗钱。

It's a process by which technologists hide ugly truths

这是一个技术人员 把丑陋真相隐藏在

inside black box algorithms

算法黑盒子中的过程，

并称之为客观；

and call them objective;

call them meritocratic.

称之为精英模式。

When they're secret, important and destructive,

当它们是秘密的， 重要的并具有破坏性的，

I've coined a term for these algorithms:

我为这些算法创造了一个术语：

"weapons of math destruction."

“杀伤性数学武器”。

(Laughter)

（笑声）

(Applause)

（鼓掌）

They're everywhere, and it's not a mistake.

它们无处不在，也不是一个错误。

These are private companies building private algorithms

这些是私有公司为了私人目的

for private ends.

建立的私有算法。

Even the ones I talked about for teachers and the public police,

甚至是我谈到的老师 与公共警察使用的（算法），

those were built by private companies

也都是由私人公司所打造的，

and sold to the government institutions.

然后卖给政府机构。

They call it their "secret sauce" --

他们称之为“秘密配方（来源）”——

that's why they can't tell us about it.

这就是他们不能告诉我们的原因。

It's also private power.

这也是私人权力。

They are profiting for wielding the authority of the inscrutable.

他们利用神秘莫测的权威来获利。

Now you might think, since all this stuff is private

你可能会想，既然所有这些都是私有的

and there's competition,

而且会有竞争，

也许自由市场会解决这个问题。

maybe the free market will solve this problem.

It won't.

然而并不会。

There's a lot of money to be made in unfairness.

在不公平的情况下， 有很多钱可以赚。

Also, we're not economic rational agents.

而且，我们不是经济理性的代理人。

We all are biased.

我们都是有偏见的。

We're all racist and bigoted in ways that we wish we weren't,

我们都是固执的种族主义者， 虽然我们希望我们不是，

in ways that we don't even know.

虽然我们甚至没有意识到。

We know this, though, in aggregate,

总的来说，我们知道这一点，

because sociologists have consistently demonstrated this

因为社会学家会一直通过这些实验

with these experiments they build,

来证明这一点，

他们发送了大量的工作申请，

where they send a bunch of applications to jobs out,

equally qualified but some have white-sounding names

都是有同样资格的候选人， 有些用白人人名，

and some have black-sounding names,

有些用黑人人名，

and it's always disappointing, the results -- always.

然而结果总是令人失望的。

So we are the ones that are biased,

所以我们是有偏见的，

我们还通过选择收集到的数据

and we are injecting those biases into the algorithms

by choosing what data to collect,

来把偏见注入到算法中，

like I chose not to think about ramen noodles --

就像我不选择去想拉面一样——

I decided it was irrelevant.

我自认为这无关紧要。

But by trusting the data that's actually picking up on past practices

但是，通过信任那些 在过去的实践中获得的数据

and by choosing the definition of success,

以及通过选择成功的定义，

how can we expect the algorithms to emerge unscathed?

我们怎么能指望算法 会是毫无瑕疵的呢？

We can't. We have to check them.

我们不能。我们必须检查。

We have to check them for fairness.

我们必须检查它们是否公平。

The good news is, we can check them for fairness.

好消息是，我们可以做到这一点。

Algorithms can be interrogated,

算法是可以被审问的，

and they will tell us the truth every time.

而且每次都能告诉我们真相。

And we can fix them. We can make them better.

然后我们可以修复它们。 我们可以让他们变得更好。

I call this an algorithmic audit,

我把它叫做算法审计，

and I'll walk you through it.

接下来我会为你们解释。

First, data integrity check.

首先，数据的完整性检查。

For the recidivism risk algorithm I talked about,

对于刚才提到过的累犯风险算法，

a data integrity check would mean we'd have to come to terms with the fact

数据的完整性检查将意味着 我们不得不接受这个事实，

that in the US, whites and blacks smoke pot at the same rate

在美国，白人和黑人 吸毒的比例是一样的，

but blacks are far more likely to be arrested --

但是黑人更有可能被逮捕——

four or five times more likely, depending on the area.

取决于区域，可能性是白人的4到5倍。

What is that bias looking like in other crime categories,

这种偏见在其他犯罪类别中 是什么样子的，

and how do we account for it?

我们又该如何解释呢？

Second, we should think about the definition of success,

其次，我们应该考虑成功的定义，

audit that.

审计它。

Remember -- with the hiring algorithm? We talked about it.

还记得我们谈论的雇佣算法吗？

Someone who stays for four years and is promoted once?

那个呆了四年的人， 然后被提升了一次？

Well, that is a successful employee,

这的确是一个成功的员工，

但这也是一名受到公司文化支持的员工。

but it's also an employee that is supported by their culture.

That said, also it can be quite biased.

也就是说， 这可能会有很大的偏差。

We need to separate those two things.

我们需要把这两件事分开。

We should look to the blind orchestra audition

我们应该去看一下乐团盲选试奏，

as an example.

举个例子。

That's where the people auditioning are behind a sheet.

这就是人们在幕后选拔乐手的地方。

What I want to think about there

我想要考虑的是

is the people who are listening have decided what's important

倾听的人已经 决定了什么是重要的，

and they've decided what's not important,

同时他们已经决定了 什么是不重要的，

他们也不会因此而分心。

and they're not getting distracted by that.

When the blind orchestra auditions started,

当乐团盲选开始时，

the number of women in orchestras went up by a factor of five.

在管弦乐队中， 女性的数量上升了5倍。

Next, we have to consider accuracy.

其次，我们必须考虑准确性。

This is where the value-added model for teachers would fail immediately.

这就是针对教师的增值模型 立刻失效的地方。

No algorithm is perfect, of course,

当然，没有一个算法是完美的，

so we have to consider the errors of every algorithm.

所以我们要考虑每一个算法的误差。

How often are there errors, and for whom does this model fail?

出现错误的频率有多高， 让这个模型失败的对象是谁？

What is the cost of that failure?

失败的代价是什么？

And finally, we have to consider

最后，我们必须考虑

the long-term effects of algorithms,

这个算法的长期效果，

the feedback loops that are engendering.

与正在产生的反馈循环。

That sounds abstract,

这听起来很抽象，

but imagine if Facebook engineers had considered that

但是想象一下 如果脸书的工程师们之前考虑过，

before they decided to show us only things that our friends had posted.

并决定只向我们展示 我们朋友所发布的东西。

I have two more messages, one for the data scientists out there.

我还有两条建议， 一条是给数据科学家的。

Data scientists: we should not be the arbiters of truth.

数据科学家们：我们不应该 成为真相的仲裁者。

We should be translators of ethical discussions that happen

我们应该成为大社会中 所发生的道德讨论的

in larger society.

翻译者。

(Applause)

（掌声）

And the rest of you,

然后剩下的人，

the non-data scientists:

非数据科学家们：

this is not a math test.

这不是一个数学测试。

This is a political fight.

这是一场政治斗争。

We need to demand accountability for our algorithmic overlords.

我们应该要求我们的 算法霸主承担问责。

(Applause)

（掌声）

The era of blind faith in big data must end.

盲目信仰大数据的时代必须结束。

Thank you very much.

非常感谢。

(Applause)

（掌声）