Rigorous Study Design and Transparent Reporting

In support of the NINDS mission, the Institute funds basic, translational, and clinical research. Rigor, control of bias, and transparency of reporting are important for all research and can significantly affect the quality of studies that provide the basis for large-scale therapy development programs and ultimately for clinical trials. Given the range of diseases within the NINDS mission and the breadth of methods used in tool development, discovery science, hypothesis-testing basic or applied research, and trial-enabling safety and efficacy studies, no single set of criteria can apply to all studies. Nevertheless, attention to principles of good study design and reporting transparency are essential to enable the scientific community to assess the quality of scientific findings and for peer reviewers to advise NINDS appropriately on funding decisions. See the NINDS Office of Research Quality for more information about NINDS's efforts to promote scientific rigor and transparency.

NINDS advises that investigators consider the following points, where appropriate, when describing key supporting data, preliminary studies, and proposed experiments in their grant applications. These items clarify and supplement, but do not supersede, guidance provided in the general NIH application instructions. The purpose of this list is to explicitly identify elements of rigorous and transparent experimental design that may be relevant to the NINDS research community and to emphasize NINDS's interest in scientific rigor and transparency more broadly:

Supporting Data:

  • Comprehensive summary of key relevant literature in support of or in disagreement with the hypothesis
  • Evaluation of quality, rigor, and transparency of methods and results in key supporting publications and preliminary experiments
  • Clear identification of exploratory (hypothesis-generating) and confirmatory (hypothesis-testing) components of key supporting experiments
  • Description of gaps in quality, rigor, transparency, interpretation, or other relevant areas in prior research that affect future experimental directions

Experimental Design:

  • Specific and testable hypotheses, if applicable
  • Rationale for the selected models, approaches, and endpoints, including their demonstrated validity
  • Authenticity of important experimental resources (noting that biological and chemical reagents must be described in the Authentication of Key Biological and / or Chemical Resources attachment to the application)
  • Rationale for parameters used in computational models or tool development
  • Plans for orthogonal approaches / triangulation to bolster inferences
  • Adequate and clearly defined controls and comparison groups
  • Route and timing of intervention delivery / dosing
  • Consideration of dose-dependency
  • Justification of sample size, such as power calculation
  • Use of biological replicates vs. technical replicates
  • Clear identification of planned exploratory (hypothesis-generating) and / or confirmatory (hypothesis-testing) components of the study
  • Prospective specification of statistical methods to be used in analysis and interpretation of results, including considerations for multiplicity
  • Plans for transparent reporting of all methods in enough detail for others to replicate in future work

Minimization of Experimental Bias:

  • Planned methods of blinding / masking (allocation concealment and blinded assessment of outcome) or rationale for why blinding / masking cannot be achieved
  • Strategies for treatment randomization and / or stratification
  • Defined criteria and rationale for inclusion or exclusion of samples from proposed experiments or analyses
  • Plans for reporting data missing due to attrition or exclusion
  • Plans for reporting all results regardless of outcome (negative / null and positive / non-null)

Results:

  • Plans for verification that the intervention or tool reached and engaged the intended target
  • Plans to determine robustness and reproducibility of the observed results, including any planned independent validation / replication
  • Plans to report sample size and effect size, along with a valid measure of uncertainty, for all relevant experiments
  • Plans to ensure accurate representation of all data in visual displays / figures
  • Plans to identify all experiments appropriately as exploratory / hypothesis-generating or confirmatory / hypothesis-testing
  • Plans for transparent reporting of all data and key metadata in an appropriate way that enables sharing with the community, including through recognized repositories (noting that some of these details must now be described in the Data Management and Sharing Plan within the application)
  • Plans for providing open access to tool / device design, computer code, and analysis pipelines involved in the study to the extent allowed by proprietary and privacy regulations

Interpretation:

  • Alternative interpretations of possible outcomes
  • Limitations of the experimental design
  • Potential generalizability of the results
  • Discussion of expected effect size in relation to potential clinical or biological impact, if applicable
  • Potential conflicts of interest

For more information, see:

comments
Want to contact NINDS staff?
Please visit our Find Your NINDS Program Officer page to learn more about contacting Program Officers, Grants Management Specialists, Scientific Review Officers, and Health Program Specialists.