Is there a reproducibility crisis in science? - Matt Anticole

In 2011, a team of physicists reported a startling discovery:

neutrinos traveled faster than the speed of light

by 60 billionths of a second

in their 730 kilometer trip from Geneva to a detector in Italy.

Despite six months of double checking, the bizarre discovery refused to yield.

But rather than celebrating a physics revolution,

the researchers published a cautious paper

arguing for continued research in an effort to explain the observed anomaly.

In time, the error was tracked to a single incorrectly connected fiber optic cable.

This example reminds us that real science is more than static textbooks.

Instead, researchers around the world are continuously publishing

their latest discoveries

with each paper adding to the scientific conversation.

Published studies can motivate future research,

inspire new products,

and inform government policy.

So it's important that we have confidence in the published results.

If their conclusions are wrong,

we risk time,

resources,

and even our health in the pursuit of false leads.

When findings are significant,

they are frequently double-checked by other researchers,

either by reanalyzing the data

or by redoing the entire experiment.

For example, it took repeated investigation of the CERN data

before the timing error was tracked down.

Unfortunately, there are currently neither the resources nor professional incentives

to double check the more than 1 million scientific papers published annually.

Even when papers are challenged, the results are not reassuring.

Recent studies that examined dozens of published pharmaceutical papers

managed to replicate the results of less than 25% of them.

And similar results have been found in other scientific disciplines.

There are a variety of sources for irreproducible results.

Errors could hide in their original design, execution, or analysis of the data.

Unknown factors,

such as patients' undisclosed condition in a medical study,

can produce results that are not repeatable in new test subjects.

And sometimes, the second research group can't reproduce the original results

simply because they don't know exactly what the original group did.

However, some problems might stem from systematic decisions

in how we do science.

Researchers,

the institutions that employ them,

and the scientific journals that publish findings

are expected to produce big results frequently.

Important papers can advance careers,

generate media interest,

and secure essential funding,

so there's slim motivation for researchers to challenge their own exciting results.

In addition, little incentive exists

to publish results unsupportive of the expected hypothesis.

That results in a deluge of agreement between what was expected

and what was found.

In rare occasions, this can even lead to deliberate fabrication,

such as in 2013, when a researcher spiked rabbit blood with human blood

to give false evidence that his HIV vaccine was working.

The publish or perish mindset

can also compromise academic journals' traditional peer-review processes

which are safety checks

where experts examine submitted papers for potential shortcomings.

The current system,

which might involve only one or two reviewers,

can be woefully ineffective.

That was demonstrated in a 1998 study

where eight weaknesses were deliberately inserted into papers,

but only around 25% were caught upon review.

Many scientists are working toward improving reproducibility in their fields.

There's a push to make researchers raw data,

experimental procedures,

and analytical techniques more openly available

in order to ease replication efforts.

The peer review process can also be strengthened

to more efficiently weed out weak papers prior to publication.

And we could temper the pressure to find big results

by publishing more papers that fail to confirm the original hypothesis,

an event that happens far more than current scientific literature suggests.

Science always has, and always will, encounter some false starts

as part of the collective acquisition of new knowledge.

Finding ways to improve the reproducibility of our results

can help us weed out those false starts more effectively,

keeping us moving steadily toward exciting new discoveries.