Reproducibility trial publishes two conclusions for one paper

How deeply an anaesthetist should sedate an elderly person when they have surgery is a controversial issue, because some studies link stronger doses of anaesthetic with earlier deaths. So it should reassure clinicians to see a study¹ in the British Journal of Anaesthesia that investigates and rules out such a link — the published paper’s discussion section says so explicitly: “These results are reassuring.”

Or are they? Another paper² in the journal analyses the same results and reaches a different conclusion about death rates. It says the trial didn’t include enough patients to reach that conclusion — or any conclusion — on mortality.

The opposing takes on the mortality link — a secondary conclusion of the study — are the result of an unusual peer-review experiment at the journal to tackle reproducibility of results in the field. In recent years, uncertainties over the reliability of studies have plagued anaesthetics research, fuelled by high-profile cases of fraud. That’s a problem, because such studies influence clinical practice and can have serious and immediate implications for patients.

So, for some papers, the British Journal of Anaesthesia is now asking an independent expert to write their own discussion of the study². Unlike conventional peer reviewers, they look only at the methods and results sections and are blinded to the paper’s conclusions³. The two discussions sections are published together, with similarities and differences highlighted.

It’s an approach that some reproducibility experts welcome and say other fields should copy. Efforts to improve reproducibility have so far focused on methods and results, and need to extend to inferences and conclusions, says John Ioannidis, one of the authors of the independent discussion and a long-standing advocate for better reproducibility in science, based at Stanford University in Palo Alto, California. “Out of very similar results with very similar methods people can make inferences or create narratives or tell stories that are very different,” he says. Independent discussion authors are free of “any allegiance bias, conflicts or any reason to favour one result or one interpretation”.

Spin and bias

The move is intended to address the “over-interpretation, spin and subjective bias” that often plague the discussion sections of academic papers, says Hugh Hemmings, editor of the British Journal of Anaesthesiaand a neuropharmacologist at Weill Cornell Medical College in New York City.

“The power of this approach will be when there is disagreement and it’s not clear who is right.” The treatment is reserved for studies in contentious or high-profile and policy-relevant areas, says Hemmings, because those studies are influential in the literature and can see their conclusions repeated and quoted.

At present, critiques of papers in the journal can appear weeks or months after publication, as guest editorials for example. By publishing the independent discussion at the same time as the peer-reviewed original, the journal hopes to accelerate the self-correcting nature of the literature. “If independent discussion authors find a fatal flaw, then we’ll have a bit of a problem. But it won’t be the first time,” says Hemmings.

The original paper’s lead author praises the approach. “I think it’s brilliant,” says Frederick Sieber, a researcher in anaesthesiology and critical-care medicine at Johns Hopkins Bayview Medical Center in Baltimore. “We’re all biased and this gives a second pair of eyes.”

In agreement

Having seen the independent discussion, Sieber agrees that the study was not big enough to robustly measure the link to mortality. “Everything they said is valid.” The original paper’s main conclusions still stand, he says, because its main goal was to report the impact of the depth of sedation on delirium, not death. The independent discussion agrees that the delirium data and conclusions are valid, because the number of patients required to test the link is smaller.

Not everyone sees value in the additional step. In an editorial published in the journal⁴, Robert Sneyd, dean of the Plymouth University Peninsula Schools of Medicine and Dentistry, UK, warns that independent discussion sections will inevitably draw on the same people who are already asked to review papers. It risks “flogging the same pool of reviewers harder or (implausibly) recruiting fresh blood,” he writes. A better solution is to enforce existing rules, such as guidelines to authors, he says — for example to make clear a study’s possible weaknesses and to avoid speculation.

Hemmings says that his journal has at least one more independent discussion lined up, and that he will continue with the idea as long as people find it useful. “It may generate so much controversy that I can’t continue to do it.”