The method that can "prove" almost anything - James A. Smith

翻译人员: Xiwen Zhu 校对人员: Yanyan Hong

In 2011, a group of researchers conducted a scientific study

2011 年，一组研究人员 进行了一项不可能找到结论的

to find an impossible result:

科学研究：

that listening to certain songs can make you younger.

听某些歌曲可以让你变得更年轻。

Their study involved real people, truthfully reported data,

这项研究邀请真人参与， 采用真实有效的数据，

and commonplace statistical analyses.

以及常见的统计分析。

So how did they do it?

那么，他们是怎么做到的呢？

The answer lies in a statistical method scientists often use

答案就在科学家们 常用的一种统计方法，

to try to figure out whether their results mean something or if they’re random noise.

用来检测他们的数据 是有意义的还是随机的。

In fact, the whole point of the music study

事实上，这项关于音乐的研究的就是

was to point out ways this method can be misused.

为了指出这个统计方法 有哪些被滥用的途径。

A famous thought experiment explains the method:

一个著名的思想实验可以 说明这个统计方法：

there are eight cups of tea,

假如你有 8 杯茶，

four with the milk added first, and four with the tea added first.

其中 4 杯先加了牛奶， 另外 4 杯先加了茶。

A participant must determine which are which according to taste.

参与者们必须根据口味 将这 8 杯茶分成两组，

There are 70 different ways the cups can be sorted into two groups of four,

总共有 70 个不同的方式 可以将杯子分为两组，每组 4 个，

and only one is correct.

而只有一个方式是正确的。

So, can she taste the difference?

那么，她可以尝出差异吗？

That’s our research question.

这就是我们的研究问题。

To analyze her choices, we define what’s called a null hypothesis:

为了分析她的选择， 我们需要建立一个原假设：

that she can’t distinguish the teas.

她不能分辨出这 8 杯茶。

If she can’t distinguish the teas,

假如她分辨不出这些茶，

she’ll still get the right answer 1 in 70 times by chance.

她仍然可以在 70 次中做一次正确选择。

1 in 70 is roughly .014.

70 分之 1 大概是 0.014 的概率。

That single number is called a p-value.

这个数字可以称之为 P 值。

In many fields, a p-value of .05 or below is considered statistically significant,

许多领域中，当 P 值等于或小于0.05时， 可以认为在统计学上具有显著性，

meaning there’s enough evidence to reject the null hypothesis.

这意味着有着足够的证据可以 拒绝我们的零假设。

Based on a p-value of .014,

基于 0.014 的 P 值，

they’d rule out the null hypothesis that she can’t distinguish the teas.

他们可以推翻她无法辨别 这 8 杯茶的原假设。

Though p-values are commonly used by both researchers and journals

虽然研究人员和期刊常使用 P 值

to evaluate scientific results,

来鉴定试验结果，

they’re really confusing, even for many scientists.

但是甚至对许多科学家来说， P 值非常难理解。

That’s partly because all a p-value actually tells us

这是因为在原假设是正确的情况下，

is the probability of getting a certain result,

P 值其实只能告诉我们

assuming the null hypothesis is true.

得到某种结果的概率。

So if she correctly sorts the teas,

所以假如她正确地分类出这 8 杯茶，

the p-value is the probability of her doing so

P 值代表的是 在假设她无法辨别这些茶的情况下

assuming she can’t tell the difference.

她能正确地分类的概率。

But the reverse isn’t true:

但反过来就不成立了：

the p-value doesn’t tell us the probability

P 值不能代表她能够尝出来

that she can taste the difference,

不同味道的概率，

which is what we’re trying to find out.

但这就是我们想要找出的。

So if a p-value doesn’t answer the research question,

所以当 P 值不能解答 我们的研究问题时，

why does the scientific community use it?

为什么科学界还在使用 P 值呢？

Well, because even though a p-value doesn’t directly state the probability

这是因为，虽然 P 值不能直接说明

that the results are due to random chance,

实验结果是随机的，

it usually gives a pretty reliable indication.

但它是一个可靠的指示。

At least, it does when used correctly.

至少，当它在正确使用时，确实如此。

And that’s where many researchers, and even whole fields,

这就是许多研究人员， 甚至整个领域

have run into trouble.

会出错的地方。

Most real studies are more complex than the tea experiment.

大多数真正的研究 都比 8 杯茶实验更加复杂，

Scientists can test their research question in multiple ways,

科学家可以使用许多不同的方法 来测试他们的研究问题，

and some of these tests might produce a statistically significant result,

其中有一些方法可能会产生 具有统计显著性的结果，

while others don’t.

但有些方法就不会。

It might seem like a good idea to test every possibility.

似乎测试每一种可能性 是一个很好的主意。

But it’s not, because with each additional test,

但是并不然， 因为每一个附加的测试

the chance of a false positive increases.

都会带来增加误报的可能性。

Searching for a low p-value, and then presenting only that analysis,

寻找低 P 值， 然后只展示这部分的分析，

is often called p-hacking.

通常称为 P 值操纵。

It’s like throwing darts until you hit a bullseye

这就好比一直扔飞镖， 直到有一个击中靶心，

and then saying you only threw the dart that hit the bull’s eye.

然后宣称你只扔了那一个 正中靶心的飞镖。

This is exactly what the music researchers did.

这正是这项音乐研究人员做的，

They played three groups of participants each a different song

他们为三组不同的参与者 放了三个不同的歌曲，

and collected lots of information about them.

然后收集了他们的大量信息。

The analysis they published included only two out of the three groups.

他们发表的分析报告 只包括了三组中的两组。

Of all the information they collected,

在所有收集的信息中，

their analysis only used participants’ fathers’ age—

他们的分析只使用了 参与者父亲的年龄

to “control for variation in baseline age across participants.”

目的是为了 “控制参与者的年龄基线”，

They also paused their experiment after every ten participants,

他们还在每 10 个参与者后暂停实验，

and continued if the p-value was above .05,

然后如果 P 值大于 0.05， 他们会继续试验，

but stopped when it dipped below .05.

但假如 P 值开始下降到低于 0.05， 他们会停止实验。

They found that participants who heard one song were 1.5 years younger

他们发现听到某一首歌的参与者 会比听到另一首歌的

than those who heard the other song, with a p-value of .04.

年轻 1.5 岁，P 值为 0.04。

Usually it’s much tougher to spot p-hacking,

通常很难发现是否存在 P 值操控，

because we don’t know the results are impossible:

因为我们不知道哪些结论 是不可能的：

the whole point of doing experiments is to learn something new.

实验的目的就是探索新的知识。

Fortunately, there’s a simple way to make p-values more reliable:

幸运的是，有许多容易的方法 可以让 P 值更加地可靠：

pre-registering a detailed plan for the experiment and analysis

预先记录实验及分析的详细计划，

beforehand that others can check,

以便他人能够核查，

so researchers can’t keep trying different analyses

来确保在研究员得到重要结果前，

until they find a significant result.

不会尝试更改分析方式。

And, in the true spirit of scientific inquiry,

本着真正的科学探究精神，

there’s even a new field that’s basically science doing science on itself:

甚至还有一个新领域出现， 那就是用科学研究科学：

studying scientific practices in order to improve them.

研究不同的科学实践， 以改进它们。