Improving the Quality of NINDS-Supported Research through Rigorous Study Design and Transparent Reporting

The mission of the National Institute of Neurological Disorders and Stroke (NINDS) is to seek fundamental knowledge about the brain and nervous system and to use that knowledge to reduce the burden of neurological disease for all people. In support of this mission, NINDS funds basic, translational, and clinical research. Rigor, control of bias, and transparency of reporting are important for all research and can significantly affect the quality of studies that provide the basis for large-scale therapy development programs and ultimately for clinical trials. Given the range of diseases within the NINDS mission and the breadth of methods used in tool development, discovery science, hypothesis-testing basic or applied research, and trial-enabling safety and efficacy studies, no single set of criteria can apply to all studies. Nevertheless, attention to principles of good study design and reporting transparency are essential to enable the scientific community to assess the quality of scientific findings and for peer reviewers to advise NINDS appropriately on funding decisions.

For this reason, NINDS advises that investigators consider the following points, where appropriate, when describing key supporting data, preliminary studies, and proposed experiments in their grant applications. These items clarify and supplement, but do not supersede, guidance provided in the general NIH application instructions. The purpose of this list is to explicitly identify elements of rigorous and transparent experimental design that may be relevant to the NINDS research community and to emphasize NINDS's interest in scientific rigor and transparency more broadly:

Supporting Data:

  • Summary of key relevant literature in support of or in disagreement with the hypothesis
  • Evaluation of quality, rigor, and transparency of methods and results in key supporting publications and preliminary experiments
  • Clear identification of exploratory (hypothesis-generating) and confirmatory (hypothesis-testing) components of key supporting experiments

Experimental Design:

  • Rationale for the selected models and endpoints
  • Authenticity of important experimental resources (noting that biological and chemical reagents must be described in the Authentication of Key Biological and / or Chemical Resources attachment to the application)
  • Rationale for parameters used in computational models or tool development
  • Adequate and clearly defined controls and comparison groups
  • Route and timing of intervention delivery / dosing
  • Consideration of dose-dependency
  • Justification of sample size, such as power calculation
  • Use of biological replicates vs. technical replicates
  • Plans for transparent reporting of all methods in enough detail for others to replicate in future work
  • Clear identification of planned exploratory (hypothesis-generating) and / or confirmatory (hypothesis-testing) components of the study
  • Prospective specification of statistical methods to be used in analysis and interpretation of results, including considerations for multiplicity

Minimization of Bias:

  • Planned methods of blinding (allocation concealment and blinded assessment of outcome) or rationale for why blinding cannot be achieved
  • Strategies for treatment randomization and / or stratification
  • Defined criteria and rationale for inclusion or exclusion of samples from proposed experiments or analyses
  • Plans for reporting data missing due to attrition or exclusion
  • Plans for reporting all results (negative / null and positive / non-null)

Results:

  • Plans for verification that the intervention or tool reached and engaged the intended target
  • Plans to determine robustness and reproducibility of the observed results, including any planned independent validation / replication
  • Plans to report sample size and effect size, along with a valid measure of uncertainty, for all relevant experiments
  • Plans for transparent reporting of all data and key metadata in an appropriate way that enables sharing with the community, including through recognized repositories (noting that some of these details must now be descibed in the Data Management and Sharing Plan within the application)
  • Plans for providing open access to tool / device design, computer code, and analysis pipelines involved in the study to the extent allowed by proprietary and privacy regulations

Interpretation:

  • Alternative interpretations of possible outcomes
  • Limitations of the experimental design
  • Potential generalizability of the results
  • Discussion of expected effect size in relation to potential clinical or biological impact, if applicable
  • Potential conflicts of interest

For more information, see:

NIH Grants & Funding: Enhancing Reproducibility through Rigor and Transparency

NINDS Office of Research Quality