{"id":1480,"date":"2018-08-28T07:33:54","date_gmt":"2018-08-28T07:33:54","guid":{"rendered":"http:\/\/163.180.4.222\/lab\/?p=1480"},"modified":"2019-10-15T19:26:34","modified_gmt":"2019-10-15T10:26:34","slug":"high-profile-journals-put-to-reproducibility-test","status":"publish","type":"post","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=1480","title":{"rendered":"High-profile journals put to reproducibility test"},"content":{"rendered":"

 <\/p>\n

 <\/p>\n

(\uc6d0\ubb38<\/a>)<\/p>\n

 <\/p>\n

 <\/p>\n

Researchers replicated 62% of social-behaviour findings published in\u00a0Science<\/em>\u00a0and\u00a0Nature<\/em>\u00a0\u2014 a result matched almost exactly by a prediction market.<\/h5>\n

 <\/p>\n

 <\/p>\n

\n

A reproducibility effort has put high-profile journals under the spotlight by trying to replicate a slew of social-science results. In the work, published on 27 August in\u00a0Nature Human Behaviour<\/i>1<\/a><\/sup>, researchers attempted to reproduce 21 social-science results reported in\u00a0Science<\/i>and\u00a0Nature<\/i>\u00a0between 2010 and 2015 and were able to reproduce 62% of the findings. That\u2019s about twice the rate achieved by an earlier effort2<\/a><\/sup>that examined the psychology literature more generally, but the latest result still raises questions about two out of every five papers studied.<\/p>\n

Reproducibility of published work has been tested before2<\/a><\/sup>,<\/sup>3<\/a><\/sup>, but this is the first such effort that focuses on the top journals. \u201cPutting the magnifying lens on high-impact journals is very necessary\u201d, says Sarahanne Field, who studies meta-research at the University of Groningen in the Netherlands. \u201cWe assume that the quality of work in such outlets is always top-notch,\u201d she says \u2014 but if it isn\u2019t reproducible, \u201cwe need to re-evaluate how we see high-impact journals and what they offer.\u201d<\/p>\n

The researchers \u2014 led by Brian Nosek of the University of Virginia in Charlottesville, who directed a\u00a0previous effort to replicate 100 psychology studies<\/a>\u00a0\u2014 also created a \u2018prediction market\u2019 in which experts could bet on how reproducible a claim was likely to be. The market generated an overall replication rate very close to that observed in the study.<\/p>\n

 <\/p>\n

Main findings<\/h2>\n

To go about reproducing the studies, the researchers selected the key finding of each paper and sought to test them using protocols checked (in all cases bar one) by the original authors. They also increased the sample sizes compared with those used in the original studies, on average by a factor of five, to improve the confidence in the outcomes.<\/p>\n

The team found a statistically significant effect in the same direction as was observed in the original study, for 13 of the 21 papers. But the strength of the effect was often smaller than what was originally reported: by about 50%, on average.<\/p>\n

The claims tested ranged from a link between analytical thinking and religious scepticism to how \u201cwriting about testing worries boosts exam performance in the classroom\u201d.<\/p>\n

Nosek says that high-profile findings like often receive substantial media interest. The study on exam performance4<\/a><\/sup>, for example, \u201chas a lot of potential implications for stress and coping strategies in high-stakes situations\u201d, he says. But the effect described was one of the findings that the researchers could not replicate.<\/p>\n

Nosek suggests that the larger effect sizes reported in the original papers might have been partly due to the smaller sample sizes used. In that situation, \u201cstudies that obtain a significant result are likely to be exaggerations of the actual effect size\u201d, he says.<\/p>\n

The differences in effect size might also be the result of a publication bias, says cognitive scientist David Rand at the Massachusetts Institute of Technology in Cambridge: journals are more inclined to publish larger effects, and so will select for such studies.<\/p>\n

Rand cautions that a failure to replicate should not be seen as an automatic invalidation of the original study \u2014 who, after all, is to say which of the two outcomes is correct?<\/p>\n

Besides, he says, meta-analyses often find a range of effect sizes including many non-significant results, even where there is an overall significant effect. For some of the papers that could not be replicated,\u00a0Nature Human Behaviour<\/i>\u00a0has published accompanying commentaries from the original authors suggesting possible reasons for the discrepancies.<\/p>\n

 <\/p>\n

Betting market<\/h2>\n

To create the prediction market, for each paper included in their study, Nosek\u2019s team assembled panels of up to 80 researchers, mostly psychologists and economists. Each participant read the study and could trade \u201cshares\u201d in the reliability of the reported result. The market predictions correlated well with the actual results, and generated an expected overall replication rate nearly identical to the one observed.<\/p>\n

This result of the prediction market \u2014\u00a0a concept tested before in the context of reproducibility<\/a>\u00a0\u2014 suggest that \u201cscientists are good assayers of the empirical reality of the world\u201d, says Nicholas Christakis of Yale University in New Haven, Connecticut, who co-authored one of the studies that the new work was able to replicate.<\/p>\n

Nosek says that participants in these markets often based their assessments on the quality of the statistical evidence and the plausibility of the original result. \u201cIf the original result was surprising, participants report having a sense that it is less likely to be true,\u201d he says. \u201cHence the aphorism that extraordinary claims require extraordinary evidence.\u201d<\/p>\n

Researchers can take several steps to improve rates of reproducibility, says Nosek. In particular, researchers could be more open about their procedures and data, and should clearly state their aims and hypotheses when pre-registering experiments \u2014 a practice that is rising rapidly in the social-behavioural sciences. \u201cResearchers are taking reproducibility seriously and looking for ways to improve the credibility and transparency of their claims\u201d, Nosek says. \u201cIt\u2019s a very exciting time.\u201d<\/p>\n

Authors could also be required to conduct replications of their own key experiments, and include those replications with the initial submission, says Rand.<\/p>\n

Richard Klein of the Universit\u00e9 Grenoble Alpes in France says that journals, too, can take action. For example, they could enforce rules about data sharing, require minimum standards of statistical power, require greater transparency and provide incentives to foster that transparency.<\/p>\n

Spokespersons for\u00a0Nature<\/i>\u00a0and\u00a0Science<\/i>\u00a0say that both journals are trying to encourage authors to explain their methods as fully as possible, to aid evaluation and replication of the work reported. (Nature<\/i>\u2019s news team is editorially independent of its journal team.)<\/p>\n

Klein thinks that a different research culture might also be needed. \u201cThe emphasis on novel, surprising findings is great in theory, but in practice it creates publication incentives that don\u2019t match the incremental, careful way science usually works,\u201d he says. He thinks that bottom-up approaches to improve reproducibility, such as informal exchanges of best practice between labs, rather than top-down directives, might prove the most effective solution. \u201cThe vast majority of scientists want to do good science and to publish things they think are true,\u201d he says.<\/p>\n

 <\/p>\n

 <\/p>\n<\/div>\n

doi: 10.1038\/d41586-018-06075-z<\/div>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

    (\uc6d0\ubb38)     Researchers replicated 62% of social-behaviour findings published in\u00a0Science\u00a0and\u00a0Nature\u00a0\u2014 a result matched almost exactly by a prediction market.     A(more…)<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[32,29,30],"tags":[6,3,4],"class_list":["post-1480","post","type-post","status-publish","format-standard","hentry","category-essays-on-science","category-lets-do-science","category-recent-science-news","tag-essays-on-science","tag-lets-do-science","tag-recent-science-news"],"aioseo_notices":[],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack-related-posts":[{"id":3493,"url":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=3493","url_meta":{"origin":1480,"position":0},"title":"Brazilian biomedical science faces reproducibility test","author":"biochemistry","date":"May 10, 2019","format":false,"excerpt":"\u00a0 \u00a0 Researchers at more than 60 Brazilian labs will assess the replicability of research by their country\u2019s scientists. \u00a0 An ambitious project to test the reproducibility of biomedical experiments by Brazilian scientists is about to get under way. The Brazilian Reproducibility Initiative was launched last year by researchers at\u2026","rel":"","context":"In "Let's Do Biology!"","block_context":{"text":"Let's Do Biology!","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?cat=33"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3627,"url":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=3627","url_meta":{"origin":1480,"position":1},"title":"Reproducibility trial publishes two conclusions\ufeff for one paper","author":"biochemistry","date":"June 5, 2019","format":false,"excerpt":"\u00a0 \u00a0 The\u00a0British\u00a0Journal of Anaesthesia\u2019s unusual experiment is designed to broaden replicability efforts beyond just methods and results. \u00a0 \u00a0 Anaesthesia has been linked to delirium and death in older patients.Credit: BSIP\/UIG via Getty \u00a0 \u00a0 How deeply an anaesthetist should sedate an elderly person when they have surgery is\u2026","rel":"","context":"In "Essays on Science"","block_context":{"text":"Essays on Science","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?cat=32"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2237,"url":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=2237","url_meta":{"origin":1480,"position":2},"title":"Industry is more alarmed about reproducibility than academia","author":"biochemistry","date":"December 3, 2018","format":false,"excerpt":"\u00a0 \u00a0 The reproducibility crisis in biomedical science seems to have alarmed industry more than the academic community (see\u00a0C. G. Begley and L. M. Ellis\u00a0Nature\u00a0483, 531\u2013533; 2012). In our view, this is because they have different yardsticks for success in research. Despite the advent of important new therapeutics, the number\u2026","rel":"","context":"In "Essays on Science"","block_context":{"text":"Essays on Science","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?cat=32"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1445,"url":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=1445","url_meta":{"origin":1480,"position":3},"title":"No more excuses for non-reproducible methods","author":"biochemistry","date":"August 24, 2018","format":false,"excerpt":"\u00a0 \u00a0 (\uc6d0\ubb38) \u00a0 \u00a0 Online technologies make it easy to share precise experimental protocols \u2014 and doing so is essential to modern science, says Lenny Teytelman. \u00a0 \u00a0 Here\u2019s a one-two punch to spark camaraderie among scientists. First, ask: \u201cHow long did it take to get your PhD?\u201d Then\u2026","rel":"","context":"In "Essays on Science"","block_context":{"text":"Essays on Science","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?cat=32"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3166,"url":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=3166","url_meta":{"origin":1480,"position":4},"title":"Duke University\u2019s huge misconduct fine is a reminder to reward rigour","author":"biochemistry","date":"April 3, 2019","format":false,"excerpt":"\u00a0 \u00a0 US$112.5-million settlement concerning fraudulent data is a casualty of a culture that prizes impact over robustness, says Arturo Casadevall. \u00a0 \u00a0 Last week, Duke University announced it would pay the US government US$112.5 million to settle claims that fraudulent data were used in dozens of research-grant applications. This\u2026","rel":"","context":"In "Essays on Science"","block_context":{"text":"Essays on Science","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?cat=32"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2675,"url":"https:\/\/biochemistry.khu.ac.kr\/lab\/?p=2675","url_meta":{"origin":1480,"position":5},"title":"Multi-omic measurements of heterogeneity in HeLa cells across laboratories","author":"biochemistry","date":"February 19, 2019","format":false,"excerpt":"\u00a0 \u00a0 Abstract \u00a0 Reproducibility in research can be compromised by both biological and technical variation, but most of the focus is on removing the latter. Here we investigate the effects of biological variation in HeLa cell lines using a systems-wide approach. We determine the degree of molecular and phenotypic\u2026","rel":"","context":"In "Let's Do Biology!"","block_context":{"text":"Let's Do Biology!","link":"https:\/\/biochemistry.khu.ac.kr\/lab\/?cat=33"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_sharing_enabled":false,"jetpack_shortlink":"https:\/\/wp.me\/p9Xo1j-nS","_links":{"self":[{"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=\/wp\/v2\/posts\/1480","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1480"}],"version-history":[{"count":1,"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=\/wp\/v2\/posts\/1480\/revisions"}],"predecessor-version":[{"id":4418,"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=\/wp\/v2\/posts\/1480\/revisions\/4418"}],"wp:attachment":[{"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1480"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1480"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/biochemistry.khu.ac.kr\/lab\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1480"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}