The method that can "prove" almost anything - James A. Smith

譯者: Lilian Chiu 審譯者: Amanda Zhu

In 2011, a group of researchers conducted a scientific study

2011 年，一群研究者 進行了一項科學研究，

to find an impossible result:

其發現讓人難以置信：

that listening to certain songs can make you younger.

聆聽某些歌曲能讓你變年輕。

Their study involved real people, truthfully reported data,

他們的研究用到真人參與、 誠實回報的資料，

and commonplace statistical analyses.

以及常用的統計分析。

So how did they do it?

他們怎麼做到的？

The answer lies in a statistical method scientists often use

答案是一種統計方法，

科學家通常會用它來判別

to try to figure out whether their results mean something or if they’re random noise.

研究結果是有意義的， 或者只是隨機雜音。

In fact, the whole point of the music study

事實上，這項音樂研究的重點

was to point out ways this method can be misused.

就是要點出這個方法 可能如何被誤用。

A famous thought experiment explains the method:

有一個著名的思想實驗 就解釋了這個方法：

there are eight cups of tea,

有八杯茶，

four with the milk added first, and four with the tea added first.

其中四杯先加牛奶，

另外四杯先加茶。

A participant must determine which are which according to taste.

受試者要根據味道 來判斷哪一杯是哪一種。

There are 70 different ways the cups can be sorted into two groups of four,

將任意四杯分成一組， 一共會有七十種組合，

and only one is correct.

其中只有一種是正確的。

So, can she taste the difference?

我們這項研究的問題是

That’s our research question.

「她能嚐出差異嗎？」

To analyze her choices, we define what’s called a null hypothesis:

為了分析各種選擇，

我們要先設定所謂的虛無假說：

that she can’t distinguish the teas.

她無法分辨。

If she can’t distinguish the teas,

如果她無法分辨，

she’ll still get the right answer 1 in 70 times by chance.

她仍然有答對的可能，

猜對的機率有七十分之一。

1 in 70 is roughly .014.

七十分之一約為 0.014。

That single number is called a p-value.

這個數字叫做 p 值。

In many fields, a p-value of .05 or below is considered statistically significant,

在許多領域中，

等於或小於 0.05 的 p 值

被認為具有統計顯著性，

meaning there’s enough evidence to reject the null hypothesis.

意即已有證據足以摒棄這個虛無假設。

Based on a p-value of .014,

因為這個研究的 p 值為 0.014，

they’d rule out the null hypothesis that she can’t distinguish the teas.

他們就會將「她無法分辨」的 虛無假說排除。

Though p-values are commonly used by both researchers and journals

雖然研究者和期刊都經常使用 p 值

to evaluate scientific results,

來評估科學研究結果，

they’re really confusing, even for many scientists.

但就連許多科學家 也會對 p 值感到困惑，

That’s partly because all a p-value actually tells us

部分原因是 p 值其實只是告訴我們，

is the probability of getting a certain result,

如果虛無假設是真的，

assuming the null hypothesis is true.

得到某個結果的機率有多高。

So if she correctly sorts the teas,

所以，如果她把茶正確地分類，

the p-value is the probability of her doing so

p 值就是在假設 她無法分辨的前提下

assuming she can’t tell the difference.

正確分辨的機率，

But the reverse isn’t true:

但反過來就不見得是對的：

the p-value doesn’t tell us the probability

p 值不會告訴我們 她分辨錯誤的機率，

that she can taste the difference,

which is what we’re trying to find out.

這機率才是我們想找出的答案。

So if a p-value doesn’t answer the research question,

所以，如果 p 值不能解答研究問題，

why does the scientific community use it?

為什麼仍被科學界採用？

Well, because even though a p-value doesn’t directly state the probability

因為雖然 p 值不能直接代表

隨機猜中的機率，

that the results are due to random chance,

it usually gives a pretty reliable indication.

但它通常仍然能提供蠻可靠的暗示，

At least, it does when used correctly.

至少是在正確使用的情況下。

And that’s where many researchers, and even whole fields,

這就是許多研究者，

甚至整個研究領域遇到問題的地方了。

have run into trouble.

Most real studies are more complex than the tea experiment.

大部分真正的研究 都比這個茶的實驗複雜許多。

Scientists can test their research question in multiple ways,

科學家可以用多種方式 來測試他們的研究，

and some of these tests might produce a statistically significant result,

有些測試可能會產生 具有統計顯著性的結果，

while others don’t.

有些則不會。

It might seem like a good idea to test every possibility.

測試每一種可能性似乎是個好點子，

But it’s not, because with each additional test,

但事實並非如此，

因為每增加一項測試，

the chance of a false positive increases.

結果是偽真的可能性就會增加。

Searching for a low p-value, and then presenting only that analysis,

找一個很低的 p 值，

並只呈現對應該 p 值的分析，

is often called p-hacking.

通常被稱為 p 值駭客。

It’s like throwing darts until you hit a bullseye

這就像是不斷射飛鏢， 直到命中紅心，

and then saying you only threw the dart that hit the bull’s eye.

然後宣稱你只射了 命中紅心的那個飛鏢。

This is exactly what the music researchers did.

那些聲稱音樂可以駐顏的研究者 用的就是這一招。

They played three groups of participants each a different song

針對三組受試者， 他們各播放一首不同的歌曲，

and collected lots of information about them.

接著收集許多實驗的資訊。

The analysis they published included only two out of the three groups.

他們發表的分析 只包含三組當中的兩組。

Of all the information they collected,

在他們所收集到的所有資訊中，

their analysis only used participants’ fathers’ age—

他們的分析只使用了 受試者的父親年齡——

to “control for variation in baseline age across participants.”

以「控制各受試者 基線年齡的差異」。

They also paused their experiment after every ten participants,

而且每做完十個受試者， 他們就會把實驗暫停，

and continued if the p-value was above .05,

如果 p 值高於 0.05 就會繼續，

but stopped when it dipped below .05.

若低於 0.05，就停下來。

They found that participants who heard one song were 1.5 years younger

他們發現，聽某一首歌曲的受試者

比聽另一首歌曲的受試者 還要年輕一歲半，

than those who heard the other song, with a p-value of .04.

對應的 p 值為 0.04。

Usually it’s much tougher to spot p-hacking,

一般來說，p 值駭客很難被發現，

because we don’t know the results are impossible:

因為我們不會知道結果是不可能的：

the whole point of doing experiments is to learn something new.

做實驗的目的就是想取得新知。

Fortunately, there’s a simple way to make p-values more reliable:

幸運的是，有一個簡單的方法

可以讓 p 值變得更可靠：

pre-registering a detailed plan for the experiment and analysis

事先登錄實驗及分析計畫，

beforehand that others can check,

讓他人能夠檢查，

so researchers can’t keep trying different analyses

這樣研究者就無法 不斷嘗試不同的分析，

until they find a significant result.

直到找到顯著的結果為止。

And, in the true spirit of scientific inquiry,

而且，根據真正的科學調查精神，

there’s even a new field that’s basically science doing science on itself:

甚至有一個新領域，

基本上是科學在對自己做科學：

studying scientific practices in order to improve them.

研究的是科學的研究方法，以改善它們。