By Gaelyn Lyons
On June 7th, I presented my first biomedical sciences PhD student seminar. I was excited to share the research I’ve done over the past year with my peers and obtain feedback that I can use moving forward. A week before my seminar, Dr. Ralph Keil, the biomedical sciences program director, sent me an email outlining what was required for the presentation. The one requirement that stuck out to me was that a specific example of rigor and reproducibility related to my work must be included in my presentation.
As research scientists, we strive to ensure the data we generate are reliable and repeatable. The new knowledge we gain through experiments will become the foundation of future research projects, and this buildable nature of science is why it is important for experiments to be rigorous, reproducible, and transparent.
The NIH defines scientific rigor as “the strict application of the scientific method to ensure unbiased and well-controlled experimental design, methodology, analysis, interpretation and reporting results.”1 To ensure research meets these criteria, the NIH requires detailed experimental design and transparency in their grant review process.2 The NIH enacted this policy in October 2015 to demand more rigorous scientific research.
Many past publications lack rigor, reproducibility, and transparency. The US Department of Health and Human Services (DHHS), Office of Research Integrity defines research misconduct as ‘a fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results.’ The most notorious example of research misconduct and lack of transparency is the 1998 paper published by Dr. Andrew Wakefield. The paper claimed there was a connection between the measles, mumps, and rubella (MMR) vaccine and the development of autism in children.3 It wasn’t until twelve years later that the Lancet retracted the paper after Britain’s General Medical Council discovered that Wakefield’s studies were not rigorous and were funded by lawyers that represented parents of autistic children suing vaccine manufacturers. The amount of scientific articles that have been retracted due to research misconduct has increased 10-fold since 1975.4 While this may seem alarming, intentional scientific misconduct is not necessarily to blame for the majority of irreproducible results observed.5 Other reasons include variability in complex techniques, not reporting whether the study was blinded and randomized, and not utilizing methods that minimize unconscious and unintended bias.6
In December 2021, scientists from the Center for Open Science (COS), a non-profit organization whose goal is to “increase the openness, integrity, and reproducibility of science research,” published a paper outlining their findings after attempting to replicate 193 experiments from 53 high-impact cancer biology papers.7 This project, referred to as the Reproducibility Project: Cancer Biology (Figure 1), took place between 2012 and 2020 and aimed to repeat key experiments from high-impact papers that were published between 2010 and 2012.7
However, due to challenges they experienced during experiments, they were only able to replicate 50 experiments from 23 papers with 80% of the replications showing similar results to the original papers.8 Some of these challenges included unhelpful original authors (32% of experiments), needing modifications to the original protocol (67% of experiments), and incomplete protocols in the manuscript (100% of experiments).7 The paper recommended using Registered Reports to help implement better rigor and reproducibility into published research. Registered Reports are articles that outline details of a proposed research study before any research is performed.9 Before data collection, the reports are peer-reviewed and if they meet the highly rigorous criteria, the study is provisionally approved for publication (Figure 2).9 These pre-approvals can help prevent publication bias by basing the reasoning for publication on the importance of the research question and not on whether the outcomes are “exciting.” 9 As of 2021, over 300 journals are currently using Registered Reporting during their publication process.7
In addition to implementing rigor and reproducibility policies4, the NIH developed a training course on enhancing reproducibility and transparency in research and now requires their trainees to take the course.5 In an article outlining the issue of lack of reproducibility and how the NIH is approaching them, Dr. Francis Collins, the former NIH director, discussed that some of the reasons for the lack of reproducibility in research are poor training of researchers in experimental design, increased emphasis on making provocative statements rather than presenting technical details, and publications that do not report basic elements of experimental design5. With the new mandatory training and grant policies put into place, the NIH hopes to improve scientific reproducibility in publications to come.
There are so many reasons why rigorous science is important. From being able to reproduce essential experiments to being transparent with both scientists and the public, rigorous science makes good science. If you are interested in learning more about rigor and reproducibility, you can find the NIH training courses here.
TL:DR
Scientific rigor and reproducibility are important for the foundation of scientific knowledge. The NIH has enacted policies to help encourage scientific rigor in research grants.
References
1. Rigor and Reproducibility | National Institutes of Health (NIH). https://www.nih.gov/research-training/rigor-reproducibility.
2. Know How To Address Rigor and Reproducibility in Your Next Application | NIH: National Institute of Allergy and Infectious Diseases. https://www.niaid.nih.gov/grants-contracts/rigor-and-reproducibility-forms-f.
3. Eggertson, L. Lancet retracts 12-year-old article linking autism to MMR vaccines. C. Can. Med. Assoc. J. 182, E199 (2010).
4. Hofseth, L. J. Getting rigorous with scientific rigor. Carcinogenesis 39, 21 (2018).
5. Collins, F. S. & Tabak, L. A. Policy: NIH plans to enhance reproducibility. Nat. 2014 5057485 505, 612–613 (2014).
6. Hewitt, J. A., Brown, L. L., Murphy, S. J., Grieder, F. & Silberberg, S. D. Accelerating Biomedical Discoveries through Rigor and Transparency. ILAR J. 58, 115–128 (2017).
7. Errington, T. M., Denis, A., Perfito, N., Iorns, E. & Nosek, B. A. Challenges for assessing replicability in preclinical cancer biology. Elife 10, (2021).
8. Errington, T. M. et al. Investigating the replicability of preclinical cancer biology. Elife 10, (2021).
9. Registered Reports: what are they and why are they important? | Royal Society. https://royalsociety.org/blog/2016/11/registered-reports-what-are-they-and-why-are-they-important/.
Lions Talk Science 