Brazilian biomedical science faces reproducibility test

An ambitious project to test the reproducibility of biomedical experiments by Brazilian scientists is about to get under way.

The Brazilian Reproducibility Initiative was launched last year by researchers at the Federal University of Rio de Janeiro (UFRJ). Now, the first wave of reproducibility testing is set to begin in August, with help from more than 60 laboratories scattered over 43 Brazilian research centres.

The project is one of the first to test the reproducibility of scientific research from a particular country, instead of a particular field.

Participants will attempt to replicate up to 100 biomedical experiments — with each experiment tested by 3 labs¹. The team decided to take that approach, rather than trying to reproduce full studies, to broaden its coverage of the published literature and to make it easier for volunteers to participate, says project coordinator Olavo Bohrer Amaral, a physician at the UFRJ Institute of Medical Biochemistry.

“We intend to systematically assess the reproducibility of biomedical research, covering different areas of life-science research in Brazil in an open, unbiased and transparent way,” he says.

Drilling down

To determine which experiments to test, the project’s leaders examined a sample of 30,000 biomedical articles published over the past two decades. They narrowed this list down by identifying 5,000 papers in which most of the authors — including the corresponding one — were from a Brazilian institution.

From this set of studies, the team drew up a list of 10 analytical methods commonly used in Brazil — including the MTT assay to assess cells’ metabolic activity, RT-PCR to amplify and detect specific genetic sequences, and the elevated plus maze to test rodent behaviour. The researchers then randomly chose experiments for replication that use one of these techniques.

Amaral and his team expect to finish the project by 2021, with funding from the Serrapilheira Institute in Rio de Janeiro — Brazil’s first private organization dedicated to supporting basic research in the natural sciences, computer sciences, engineering and mathematics. An initial 145,000-Brazilian-real (US$37,000) grant allowed the researchers to establish the project’s general methodology, select the experiments to analyse and build a countrywide network of collaborators.

Now the project is working with participating labs to establish protocols for replication attempts, with the help of another 1,000,000-real grant from the institute, awarded in January.

Lather, rinse, repeat

The Brazilian project follows in the footsteps of several attempts to replicate scientific outcomes on a large scale. One of the first was the Reproducibility Project: Psychology, which launched in 2011. It gathered 270 scientists to replicate the results of 100 psychology articles in different journals, yielding a reproducibility rate of 36–47%². Similar initiatives in experimental economics, philosophy and social sciences arrived at replication rates ranging between 57 and 78%.

The effort’s leaders hope that it will reveal ways to predict the reproducibility of scientific studies. “It might be invaluable for future decisions on how to finance and elevate science in Brazil,” says Roger Chammas, an oncologist at University of São Paulo School of Medicine and coordinator of one of the replicating labs.

Daniel Martins-de-Souza, a biochemist at the University of Campinas in Brazil, agrees. “If the project moves forward, it may aid defining which types of studies or methods have more potential to obtain new possibilities of therapy against diseases,” he says. “It could guide the decision-making process of funding agencies.”

Others are more sceptical. Lygia da Veiga Pereira, a geneticist at the University of São Paulo, says it is too early to tell whether the project’s findings will be able to help guide future research. Still, she says, “testing how much of Brazilian science is reproducible will be a good diagnosis for us”.