Rigor Champions and Resources

Rigor Champions and Resources

Efforts to Instill the Fundamental Principles of Rigorous Research

Rigorous experimental procedures and transparent reporting of research results are vital to the continued success of the biomedical enterprise at both the preclinical and the clinical levels; therefore, NINDS convened major stakeholders in October 2018 to discuss how best to encourage rigorous biomedical research practices. The attendees discussed potential improvements to current training resources meant to instill the principles of rigorous research in current and future scientists, ideal attributes of a potential new educational resource, and cultural factors needed to ensure the success of such training. Please see the event website for more information about this workshop, including video recordings of the discussion, or the recent publication summarizing the workshop.

Rigor Champions

As described in this publication, enthusiastic individuals ("champions") who want to drive improvements in rigorous research practices, transparent reporting, and comprehensive education may come from all career stages and sectors, including undergraduate students, graduate students, postdoctoral fellows, researchers, educators, institutional leaders, journal editors, scientific societies, private industry, and funders. We encouraged champions to organize themselves into intra- and inter-institutional communities to effect change within and across scientific institutions. These communities can then share resources and best practices, propose changes to current training and research infrastructure, build new tools to support better research practices, and support rigorous research on a daily basis.

If you are interested learning more, you can join this grassroots online workspace or email us at RigorChampions@nih.gov.

Rigor Resources

In order to understand the current landscape of training in the principles of rigorous research, NINDS is gathering a list of public resources that are, or can be made, freely accessible to the scientific community and beyond. We hope that compiling these resources will help identify gaps in training and stimulate discussion about proposed improvements and the building of new resources that facilitate training in transparency and other rigorous research practices. Please peruse the resources compiled thus far below, and contact us at RigorChampions@nih.gov to let us know about other potential resources.

NINDS does not endorse any of these resources and leaves it to the scientific community to judge their quality.

Resources Table

Categories of resources listed in the table include Books and Articles, Guidelines and Protocols, Organizations and Training Programs, Software and Other Digital Resources, and Videos and Courses.

Please use the drop-down menu under "Categories" to filter results based on one of these categories or use the "Quick Search" bar on the right to enter a key word of your choice.

Resource Description Category
10 Simple Rules for Evaluating Model Credibility Communication tool by IMAG (Interagency Modeling and Analysis Group) for modelers to organize their model development process Guidelines and Protocols
2 Min Stats Short, animated videos about statistics Videos and Courses
A Call for Transparent Reporting to Optimize the Predictive Value of Preclinical Research Article in Nature listing important experimental design elements to report in preclinical research papers Books and Articles, Guidelines and Protocols
ABRF Committee on Core Rigor and Reproducibility (CCoRRe) Discussion Forum Discussion forum by a committee of The Association of Biomolecular Resource Facilities (ABRF) that focuses on building educational resources, sharing critical best practice guidelines, and providing important tools to the core community that will impact rigor, reproducibility, and transparency across the range of science and technology Organizations and Training Programs, Software and Other Digital Resources
Advice for Choosing the Right Reproducibility Tools Webinar organized by Reproducibility for Everyone (Repro4Everyone) recorded at the Society for Neuroscience (SfN) annual meeting about tools for experimental rigor and transparency Videos and Courses
Advice to Scientists: Incorporating Authentication into Everyday Culture Practice Cell culture practice guidelines by the International Cell Line Authentication Committee (ICLAC) Guidelines and Protocols
AMSTAR (A MeaSurement Tool to Assess systematic Reviews) 2 A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions Software and Other Digital Resources
Animal Study Registry Online registry for scientific studies involving animals conducted around the world Software and Other Digital Resources
Antibodypedia Portal of validated antibodies that are scored using the validation principles outlined by the International Working Group for Antibody Validation Software and Other Digital Resources
ARRIVE Guidelines Animal Research: Reporting of In Vivo Experiments (ARRIVE) guidelines are intended to improve the reporting of research using animals Guidelines and Protocols
ASA's Statement on Statistical Significance and P-Values Consensus statement by the American Statistical Association on the proper use of p-values in research Books and Articles, Guidelines and Protocols
AsPredicted Standardized pre-registration that requires what is necessary to separate exploratory from confirmatory analyses Software and Other Digital Resources
Assay Guidance Manual Free, best-practices online resource devoted to the successful development of robust, early-stage drug discovery assays Books and Articles, Guidelines and Protocols
BD2K Guide to the Fundamentals of Data Science Series Virtual lecture series by the Big Data to Knowledge (BD2K) Initiative on the data science underlying modern biomedical research Videos and Courses
British Neuroscience Association UK-based organisation representing and promoting neuroscience and neuroscientists, with a program to promote and support credibility in neuroscience Organizations and Training Programs
Capturing the Value of Biomedical Research Article in Cell describing a framework for scientific evaluation at the Sidra Medical and Research Center in Qatar Books and Articles
Cell Line Authentication Training Training in cell line authentication developed by Global Biological Standards Institute (GBSI) Videos and Courses
Center for Biomedical Research Transparency (CBMRT) Organization that facilitates transparent reporting of biomedical research Organizations and Training Programs
Center for Innovation in Science Education (R3ISE) Johns Hopkins Bloomberg School of Public Health's effort which aims to enable young practitioners to do good science and view their work through the lens of social responsibility Organizations and Training Programs
Center for Open Science (COS) Organization that works to increase openness, integrity, and reproducibility of research Organizations and Training Programs
Center for Open Science (COS) Reproducible Research Training Materials related to educational videos posted to the COS Youtube channel Videos and Courses
Center for the Improvement of Mentored Experiences in Research (CIMER) Organization that develops programs including Entering Research with undergraduate and graduate trainee curricular materials and Entering Mentoring curriculum for research mentors Organizations and Training Programs
Common Online Data Analysis Platform (CODAP) Open-source software for dynamic data exploration in a pre-college educational setting Software and Other Digital Resources
Complexity Explorer Online courses and educational materials about complexity science produced by the Santa Fe Institute Videos and Courses
CONSORT (CONsolidated Standards of Reporting Trials) 2010 Guidelines developed through collaboration and consensus to improve the reporting of parallel-group randomized controlled trials (RCTs) Guidelines and Protocols
COSMIN (COnsensus-based Standards for the selection of health Measurement INstruments) An initiative to develop methodology and practical tools for selecting the most suitable outcome measurement instrument Organizations and Training Programs, Software and Other Digital Resources
Cross Validated A question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization Software and Other Digital Resources
DAGitty Browser-based environment for creating, editing, and analyzing causal diagrams for minimizing bias in empirical studies in epidemiology and other disciplines Software and Other Digital Resources
Data Organization by Karl Broman Short primer from Karl Broman on how to organize data in spreadsheets prior to analysis Guidelines and Protocols
DataONE (Data Observation Network for Earth) Best Practices Database of recommendations on how to effectively work with data through all stages of the data lifecycle Guidelines and Protocols, Software and Other Digital Resources
Digital Curation Centre (DCC) Center providing expert advice and practical help on how to store, manage, protect and share digital research data Organizations and Training Programs
DMPTool Online tool to help build data management plans (DMPs) to facilitate research Software and Other Digital Resources
Educational Resources to Make the Best Out of the Corona Break List of resources compiled by the Berlin Institute of Health to improve scientific practice and professional development Organizations and Training Programs, Software and Other Digital Resources
Electronic Lab Notebooks at HMS Comparison grid hosted by Harvard Medical School showing electronic lab notebook (ELN) features Software and Other Digital Resources
Enhancing Reproducibility Through Rigor and Transparency Guidance for addressing rigor in NIH grant applications and progress reports Guidelines and Protocols
Equator Network Library for Health Research Reporting Collection of guidelines and policy documents related to transparent health research reporting Guidelines and Protocols, Software and Other Digital Resources
Estimation for Better Inference in Neuroscience Article describing the estimation approach to inference, which emphasizes reporting effect sizes with expressions of uncertainty Books and Articles
Estimation Statistics Web application to plot experimental data from an estimation statistics perspective Software and Other Digital Resources
European College of Neuropsychopharmacology (ECNP) Preclinical Data Forum Network Communication platform for dialog related to modern issues in preclinical research and for preclinical data sharing with the goal of enhancing data utility for clinical R&D Organizations and Training Programs
European Quality in Preclinical Data (EQIPD) Consortium Consortium with the goal of improving preclinical data quality to accelerate the transition to clinical studies Organizations and Training Programs
Evaluation of Research Careers Fully Acknowledging Open Science Practices Recommendations by the Open Science Working Group on Rewards/Recognition for ensuring researchers in Europe are recognised and rewarded for practising open science, through the Open Science Career Assessment Matrix (OS-CAM) Books and Articles, Guidelines and Protocols
Experimental Design Assistant (EDA) Free resource developed by the National Centre for the Replacement Refinement & Reduction of Animals in Research (NC3Rs) to help diagram experimental plans Software and Other Digital Resources
Experimental Design: Top Four Strategies for Reproducible Mouse Research Key considerations for mouse research according to the Jackson Laboratory Guidelines and Protocols
FAIRsharing A curated, informative and educational resource on data and metadata standards, inter-related to databases and data policies Software and Other Digital Resources
Fiddle (FIle Drawer Data Liberation Effort) Tool to identify alternate publishing routes for well-designed experiments that are difficult to publish in traditional journals (e.g. null or neutral results, datasets) Software and Other Digital Resources
Framework for Open and Reproducible Research Training (FORRT) Pedagogical infrastructure designed to recognize and support the teaching and mentoring of open and reproducible science tenets in tandem with prototypical subject matters in higher education Organizations and Training Programs
Funding Opportunities in Rigor and Reproducibility Past and current NIH funding opportunities for efforts to improve research rigor and quality Organizations and Training Programs, Software and Other Digital Resources
Funding Promoting Open Research Crowdsourced list of funding opportunities in reproducible, open, and FAIR research Software and Other Digital Resources
Good Research Practice in Non-Clinical Pharmacology and Biomedicine Online textbook in the Handbook of Experimental Pharmacology series that describes rigorous and robust research practices Books and Articles
Guidelines for Scientific Record Keeping in the Intramural Research Program at the NIH General principles underlying record keeping that are necessary to support the conduct of good science and address needs arising from the rapid growth of alternative record keeping methods, the increasing complexity of research data formats, and the influx of scientific trainees with diverse background Books and Articles, Guidelines and Protocols
Handbook of Biological Statistics Online textbook by John H. McDonald originally developed to supplement a biological data analysis course Books and Articles
HHMI BioInteractive: Statistics and Math Statistics and math videos produced by Howard Hughes Medical Institute (HHMI) BioInteractive Videos and Courses
Hong Kong Principles for Assessing Researchers Principles for improving research through ensuring that researchers are explicitly recognized and rewarded for behavior that leads to trustworthy research Books and Articles, Guidelines and Protocols
How Scientists Fool Themselves - and How They Can Stop Editorial describing cognitive fallacies in research and how to minimize bias Books and Articles
ICPSR Guidelines for Effective Data Management Plans Inter-university Consortium for Political and Social Research (ICPSR) data repository at the University of Michigan that offers examples, guidance, and other resources in developing a data management plan Organizations and Training Programs, Software and Other Digital Resources
Improving the Trustworthiness, Usefulness, and Ethics of Biomedical Research Through an Innovative and Comprehensive Institutional Initiative Article in PLOS Biology about an effort at the Berlin Institute of Health to incentivize and support responsible practices in research, funding, and hiring Books and Articles
Improving Your Neuroscience - Society for Neuroscience (SfN) Workshop OSF project website that serves as a hub for disseminating material for the "Improving Your Science" professional development workshop series at the Annual Meeting of the Society for Neuroscience Software and Other Digital Resources, Videos and Courses
INCF TrainingSpace Online hub hosted by the International Neuroinformatics Coordinating Facility (INCF) to make neuroscience educational materials more accessible to the global neuroscience community Software and Other Digital Resources, Videos and Courses
International Neuroinformatics Coordinating Facility (INCF) Standards organization that develops, evaluates, and endorses best practices that embrace the principles of Open, FAIR, and Citable neuroscience Organizations and Training Programs
Introduction to the New Statistics Companion site for the Introduction to the New Statistics textbook Books and Articles
James Lind Library Collection of historical material to illustrate the principles underlying fair tests of treatments and their development and application over time Books and Articles
Jamovi Free and open statistical platform Software and Other Digital Resources
Journal of Biological Chemistry: Collecting and Presenting Data Guidelines for reporting data and experimental conditions as fully and transparently as possible Guidelines and Protocols
Let's Experiment: A Guide for Scientists Working at the Bench Free 6-week course to build a framework for how to design experiments in biological research Videos and Courses
Materials Design Analysis Reporting (MDAR) Checklist for Authors A minimum set of requirements in transparent reporting applicable to studies in the life sciences Guidelines and Protocols
Medical Research Council (MRC) Methodology Guidance Portal Portal developed within the NIHR-MRC Methodology Research Programme to inform best methodological practice for clinical studies Software and Other Digital Resources
Medical Research Council (MRC) Policies and Guidance for Researchers Resources and policies that all MRC-funded scientists are required to follow as a condition of their funding Guidelines and Protocols, Organizations and Training Programs
Messy Data Coalition Concord Consortium initiative to bring "messy data" into the classroom to improve pedagogy in data literacy and data fluency Organizations and Training Programs
Meta-Research Innovation Center at Stanford (METRICS) Research to action center focused on transforming research practices to improve the quality of scientific studies in biomedicine and beyond Organizations and Training Programs, Software and Other Digital Resources, Videos and Courses
Methods: Mind the Gap Webinars that explore research design, measurement, intervention, data analysis, and other methods of interest in clinical disease prevention science Videos and Courses
Modern Statistics for Modern Biology Interactive online statistics textbook by Susan Holmes and Wolfgang Huber Books and Articles
National Centre for the Replacement Refinement & Reduction of Animals in Research (NC3Rs) Organization that works to replace, reduce, or refine the use of animals in research and ensure that animal experiments are as robust and reproducible as possible Organizations and Training Programs
Nature.com Collection: Statistics for Biologists Nature's collection of statistical issues of which biologists should be aware and practical advice to help improve the rigor of their work Books and Articles
NC3Rs Experimental Design Hub Resources compiled by the National Centre for the Replacement Refinement & Reduction of Animals in Research (NC3Rs) for designing and reporting animal research Organizations and Training Programs, Software and Other Digital Resources
NeuroImaging Tools & Resources Collaboratory (NITRC) Web-based resource that shares comprehensive information on neuroinformatics software and data Software and Other Digital Resources
Next Generation Science Standards (NGSS) Research-based content standards for the K-12 (primary and secondary school) level, including education in scientific practice Guidelines and Protocols
NIGMS Clearinghouse for Training Modules to Enhance Data Reproducibility Products of NIH-funded grants and other relevant training modules developed to improve data reproducibility Videos and Courses
NIH Figshare Pilot program to test the usefulness of formally adding generalist data repositories to the NIH data ecosystem Software and Other Digital Resources
NIH Research Methods Resources NIH website that provides investigators with important research methods resources to help them satisfy new clinical trial requirements Guidelines and Protocols, Organizations and Training Programs, Videos and Courses
Open Neuroscience Network of collaborators curating open source projects related to neurosciences Software and Other Digital Resources
Open Scholarship Knowledge Base Open scholarship resources created and curated by the community Software and Other Digital Resources
Open Science Committee at the MRC Cognition and Brain Sciences Unit Members from all levels of seniority that meet termly to discuss best practices and future plans to further advance education and implementation of open science practices within and beyond the unit Organizations and Training Programs
Open Science Framework (OSF) A free and open source project management repository that supports researchers across their entire project lifecycle Organizations and Training Programs, Software and Other Digital Resources
Open Science MOOC Online courses to help equip students and researchers with the skills they need to excel in an open research environment Videos and Courses
Patient-Centered Outcomes Research Institute (PCORI) Congressionally mandated nonprofit organization that helps people make informed healthcare decisions, and improves healthcare delivery and outcomes, by producing and promoting high-integrity, evidence-based information Organizations and Training Programs
Pragmatic and Group-Randomized Trials in Public Health and Medicine A 7-part online course to help researchers design and analyze group-randomized trials (GRTs) Videos and Courses
PreclinicalTrials International prospective registry of preclinical animal study protocols Software and Other Digital Resources
PREPARE (Planning Research and Experimental Procedures on Animals: Recommendations for Excellence) Guidelines to improve the planning of research and testing using animals Guidelines and Protocols
Principles and Guidelines for Reporting Preclinical Research NIH Guidelines for transparently reporting research practices in publications Guidelines and Protocols, Organizations and Training Programs
Principles, Statistical and Computational Tools for Reproducible Data Science Online course to learn skills and tools that support data science and reproducible research Videos and Courses
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) A minimum set of items for reporting in systematic reviews and meta-analyses Guidelines and Protocols
Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience Webinar series hosted by the Society for Neuroscience (SfN) on topics of experimental rigor and transparency Videos and Courses
PROSPERO International prospective registry of systematic reviews Software and Other Digital Resources
Quality Practice in Basic Biomedical Research (QPBR) Manuals by the World Health Organization (WHO) that outline a quality management system which provides institutions and researchers with tools for the implementation and monitoring of quality practices in biomedical research Books and Articles, Guidelines and Protocols
QUEST (Quality, Ethics, Open Science, Translation) Center for Transforming Biomedical Research Center at the Berlin Institute of Health (BIH) in Germany that strives to increase the value of biomedical research at BIH and beyond Organizations and Training Programs
Register of Misidentified Cell Lines Registry maintained by the International Cell Line Authentication Committee (ICLAC) of cell lines that are known to be misidentified through cross-contamination or other mechanisms Software and Other Digital Resources
Registry of Research Data Repositories (re3data) Global registry of research data repositories that covers research data repositories from different academic disciplines Software and Other Digital Resources
ReproducibiliTea Grassroots initiative that helps young researchers create local journal clubs at their universities to discuss papers and ideas about improving science, reproducibility, and the open science movement Organizations and Training Programs
Reproducibility University of Minnesota library webpage on reproducibility resources Organizations and Training Programs, Software and Other Digital Resources
Reproducibility and Reliability of Biomedical Research: Improving Research Practice Report from a symposium held by the Academy of Medical Sciences, the Biotechnology and Biological Sciences Research Council (BBSRC), the Medical Research Council (MRC) and the Wellcome Trust in April 2015 Books and Articles
Reproducibility for Everyone Organization that compiles resources and teaching workshops to improve the reproducibility of research Organizations and Training Programs
Reproducibility in Science Github guide to enhancing reproducibility in scientific results and writing Guidelines and Protocols, Software and Other Digital Resources
Reproducibility of Research: Get Started List of courses and resources for rigor and reproducibility on the University of Utah Spencer S. Eccles Health Sciences Library website Organizations and Training Programs, Software and Other Digital Resources
Reproducible Research Course that focuses on the concepts and tools behind reporting modern data analyses in a reproducible manner Videos and Courses
Research and Data Integrity (ReaDI) Program Columbia University's compilation of online research integrity and scientific best practice resources Organizations and Training Programs, Software and Other Digital Resources
Research Culture: Setting the Right Tone Article in eLife that describes efforts to improve research culture and quality at the University of Glasgow Books and Articles
Research on Research Institute (RoRI) Collaboration to investigate how research is funded, practiced and evaluated, and in how research cultures and systems can be made more efficient, open, inclusive and impactful Organizations and Training Programs
Research Topic: Reproducibility and Rigour in Computational Neuroscience Articles by Frontiers journals about reproducibility of computational studies Books and Articles
Resources for Rigor & Transparency List of resources on the Brown University Library website for improving rigor and transparency Organizations and Training Programs, Software and Other Digital Resources
Rigorous Science: a How-To Guide Article describing a framework for rigor that includes redundant experimental design, sound statistical analysis, recognition of error, avoidance of logical fallacies, and intellectual honesty Books and Articles, Guidelines and Protocols
San Francisco Declaration on Research Assessment (DORA) Recommendations to improve the ways in which the output of scientific research is evaluated by funding agencies, academic institutions, and other parties Guidelines and Protocols
Science in Transition Dutch initiative to consider various issues related to research quality and reliability Organizations and Training Programs
Scientific Rigor Champions Online workspace for champions of rigorous research to discuss pilot initiatives, successes, methods of evaluation, and best practices Software and Other Digital Resources
Shake Up Conferences Article in Nature describing potential methods for improving rigor and transparency at conferences Books and Articles
Society for the Improvement of Psychological Science Service organization which aims to increase the quality of psychological science Organizations and Training Programs
Stanford Center for Reproducible Science Neuroimaging center that provides researchers with tools to analyze data, share data, and share workflows in ways that focus on the reproducibility of the results Organizations and Training Programs, Software and Other Digital Resources
STAR Methods Guide for Authors Cell Press guidelines for increased clarity and transparency in reporting of methods and analysis to foster improved rigor and reproducibility in research Guidelines and Protocols
Statistical Inference in the 21st Century: A World Beyond p < 0.05 Special issue of The American Statistician containing dozens of journal articles by statisticians offering advice on making scientific inferences from statistical tests Books and Articles
Statistical Tests, P Values, Confidence Intervals, and Power: a Guide to Misinterpretations An article in the European Journal of Epidemiology on common misconceptions with p values, confidence intervals, and power calculations Books and Articles
StatisticalHelp Free statistical help provided by StatsDirect Software and Other Digital Resources
Statistics Done Wrong: the Woefully Complete Guide A guide to the most popular statistical errors and slip-ups committed by scientists every day, in the lab, and in peer-reviewed journals Books and Articles
StatXFinder A decision support tool for appropriate statistical test selection Software and Other Digital Resources
Strain Background and Genetic Drift in Mice Jackson Laboratory FAQs and resources about potential genetic confounds in mouse studies Guidelines and Protocols, Organizations and Training Programs, Software and Other Digital Resources
Systematic Review Center for Laboratory Animal Experimentation (SYRCLE) Center that develops, applies, and disseminates methodology and resources for responsible and rigorous systematic reviews of animal studies Organizations and Training Programs, Software and Other Digital Resources
Teaching Reproducibility List of teaching resources at the University of Washington University Libraries to help integrate reproducibility into the curriculum Organizations and Training Programs, Videos and Courses
Ten Simple Rules for Reproducible Computational Research Article describing guidelines for making computational research more accessible Books and Articles, Guidelines and Protocols
Testing Treatments International Companion site for the Testing Treatments book that promotes critical thinking about treatment claims Books and Articles
The Many Faces of Reproducibility Project supported by the College of Liberal Arts at the University of Minnesota involving an interdisciplinary exploration of different facets of reproducibility across the social, behavioral, and natural sciences Organizations and Training Programs
The National Academies Press: Reproducibility Collection Reports by the National Academies of Sciences, Engineering, and Medicine that discuss the importance and challenges of fostering reproducibility in science Books and Articles, Guidelines and Protocols
Transparency and Openness Promotion (TOP) Guidelines Shared standards for open science practices across journals Books and Articles, Guidelines and Protocols
UCLA Institute for Digital Research and Education Seminars and course notes by the UCLA Statistical Consulting Group Organizations and Training Programs, Videos and Courses
UCSF Introduction to Biostatistics Biostatistics course materials by David Quigley for designing studies, performing essential analyses, and consuming statistical data Books and Articles, Videos and Courses
UK Network of Open Research Working Groups Action-oriented teams within higher education that are seeking to reform science to make the processes and products of research as transparent, accessible, and reproducible as possible Organizations and Training Programs
UK Reproducibility Network Peer-led consortium that investigates the factors that contribute to robust research, provides training and disseminates best practice, and works with stakeholders to ensure coordination of efforts across the sector Organizations and Training Programs
Utrecht University Open Science Programme A programme that aims to stimulate and facilitate researchers to put open science into practice Organizations and Training Programs
Whole Tale Computational tools for telling the whole tale of the research project Software and Other Digital Resources