In 2012, after more than a decade of active efforts to identify effective approaches to support replication of critical pre-clinical studies, NINDS recognized that neuroscience could benefit from an initiative to improve the quality and rigor of the research that we fund. We felt that a productive first step would be to address transparency in reporting the characteristics of experimental design and analysis that one would find necessary both to evaluate the degree of rigor employed in research and to inform attempts at reproducing research findings. Under the direction of Dr. Shai Silberberg and Director Dr. Story Landis, NINDS convened representatives from our community of scientists, funding partners, reviewers, professional and scientist organizations, and editors of journals that publish NINDS-funded research. At the Optimizing the Predictive Value of Preclinical Research workshop in 2012, meeting attendees agreed that all stakeholders share responsibility to increase transparency in reporting, and they adopted a core set of parameters that should be reported in all animal studies. These efforts were summarized in a Perspective published in Nature in October 2012.
These guidelines served as a springboard for a host of activities undertaken by NIH, scientific societies, and a number of journals. NIH embarked upon a major effort to improve rigor at every level, from training and education to grant review. NIH developed a thoughtful policy on rigor and transparency as well as training modules to enhance data reproducibility. In June 2014, NIH, Nature, and Science joined forces to organize a second workshop, with more than 30 journal editors, which led to consensus on a set of principles and guidelines for reporting preclinical research endorsed by many journals. In partnership with NIH, the Society for Neuroscience developed the series Promoting Awareness and Knowledge to Enhance Scientific Rigor in Neuroscience, available freely through “NeurOnline”.
In 2014, Drs. Steward and Balice-Gordon emphasized the importance of adopting rigorous experimental design for neuroscience in their paper “Rigor or Mortis”. Science, and one’s reputation as a scientist, lives or dies by whether the published work reflects the truth. Though sometimes circuitous, and despite the imperfections in our processes of grant funding and academic promotion, scientists and their findings eventually fall into oblivion, or rise to launch new advances, based on their proximity to the truth. The additional focus on experimental design, including sample size estimation, randomization, blinding at randomization and assessment, details on sources of biological or chemical reagents, and inclusion of sex as a variable, should improve the value of the published work. Enhancing rigor and reproducibility in experimental design and execution strengthens the foundation on which tomorrow’s advances in treating and preventing neurological disorders depend.
To encourage greater awareness of rigor and transparency in preclinical research, NINDS staff have given more than 80 presentations over the last four years at national and international funding organizations, Institutions, scientific meetings, and training courses. NINDS is continuously evaluating the effects of these activities for our research and training, and we recently launched an Office of Research Quality led by Dr. Silberberg. In this unique position, Dr. Silberberg advises leadership on issues related to rigor and transparency, leads a NINDS Scientific Rigor Working Group, represents NINDS in trans-NIH rigor initiatives, plans and evaluates rigor-related initiatives, helps incorporate rigor criteria into the NINDS review process, develops and disseminates training materials, and conducts outreach to the scientific community.
These challenges are not unique. Clinical trialists long ago incorporated very strict conditions to minimize the natural bias that surrounds the passion and hope, as well as the fame and financial benefits, associated with publishing a new therapy that benefits patients. Indeed, the Consolidated Standards of Reporting Trials (CONSORT) guidelines were designed to alleviate the problems arising from inadequate reporting of randomized controlled trials. Around the turn of the century, in response to a string of failed clinical trials testing drugs that appeared promising in animal models, the Stroke Therapy Academic Industry Roundtable (STAIR) published guidelines calling for better pre-clinical testing standards and greater transparency in scientific publications that could improve the probability of future success in human clinical trials. Physicians from the period in which “Quality Improvement” began to permeate hospital care can attest to the common finding that doctors thought they were providing superb care until faced with patient outcome data. Then, remarkable improvements resulted from instituting new procedures and tracking data over time. The adoption of recent guidelines into the culture of neuroscience should quickly result in research quality improvement.
In addition to setting out “best practices” for experimental design and transparency in reporting, the neuroscience field has gone a step further. The technologies that have recently empowered neuroscience challenge biologists among us to expand our quantitative skillset, develop needed technical expertise to utilize the new tools, and carefully analyze the plethora of data that they provide. Along these lines, the NIH BRAIN Initiative has a major focus on incorporating mathematics, bioinformatics, and computational science into the effort to understand the fundamental principles by which brain circuit activity gives rise to behavior. Statistics should not be considered an obstacle that has to be negotiated in order to publish, but rather statistical principles should be considered in the initial research design and should inform every step of the experimental process, including analyzing data, interpreting results, and forming conclusions. To this end, the NIH neuroscience institutes have recently revised their jointly sponsored training program to stress experimental design, statistical methodology, and quantitative reasoning.
There is also a need for exploring “robustness” in research findings. Are the findings substantial, and do they generalize across relevant experimental conditions? Results with highly significant p-values but small effect sizes are likely to see trouble when another lab tries to reproduce the findings. Especially when moving findings to clinical translation, a result with a small effect size that can only be reproduced under very confined conditions is unlikely to survive the tremendous variation inherent in human studies. Exploring the conditions under which a result does not stand up is important for translation, but is too rarely pursued.
Finally, and importantly, NINDS wants to state explicitly that the findings of experimentalists have revolutionized our understanding of the nervous system. Senior scientists marvel at the sophistication of today’s science compared to when they trained. The crude disease models of a few decades ago have been replaced by animal models and human derived neural cells, and these advances are producing important new insights into the causes and progression of neurological disorders. Despite our relatively incomplete knowledge base, and faced with the enormous complexity of the nervous system, the published works of the past few decades have had tangible benefits for health. New approaches for preventing cerebral palsy and stroke, and new treatments for disorders such as multiple sclerosis, epilepsy, acute ischemic stroke, Parkinson’s disease, some lysosomal storage disorders, and more recently spinal muscular atrophy, have made a tremendous difference in people’s lives. We have achieved a molecular-level knowledge of many other devastating neurological disorders, and we are now poised to develop new breakthrough treatments. Moving forward, scientific inquiry must be pursued with an exacting attention to experimental design and a focus on robust, reproducible results that is matched by transparency in reporting. NINDS has made great strides over the last five years, and we are dedicated to ensuring that the neuroscience community’s focus on these issues continues to strengthen and that our research delivers treatments to patients as quickly as possible.