Quality Assurance Guidelines

1.0. Introduction, Purpose and Background

The National Institute of Neurological Disorders and Stroke (NINDS), National Institutes of Health (NIH), currently supports over 1,000 clinical research projects. The accuracy and integrity of these studies are of paramount importance to protecting human participants and achieving the scientific objectives. The purpose of these Guidelines is to assist investigators in developing and implementing quality assurance (QA) procedures that can be integrated into their clinical research projects.

The objectives of QA procedures are to assure the accuracy and consistency of study data, from the original observations through the reporting of results and to ensure that study results are considered valid and credible within the scientific and clinical communities.

The Guidelines specify the procedures investigators should develop and implement to prevent, to detect, and to correctdata quality and integrity problems. They are presented in three formats:

  • Checklist - outline for investigators to use in planning and assessing QA procedures.
  • Guidelines - text for investigators and study staff who would like more descriptive guidance about the checklist.
  • Bibliography - suggested references about QA.

This document contains the Guidelines and Bibliography. It can be used for multi-center study clinical research projects.

2.0. Generic Quality Assurance Activities

There are some generic activities that help to ensure quality in a clinical research environment. These include the following:

  • Standard Operating Procedures (SOPs) - describe general functions of a clinical study group. The purposes of SOPs are to document key operating activities that are applicable to all studies and to ensure a consistent and comprehensive approach for multiple staff members.  SOPs are often used to train new staff members. SOPs address activities such as, but not limited to, protocol review and sign-off, IRB approval process, adverse event reporting, records storage, sample labeling and storage, and data management. SOPs are important for both coordinating centers and clinical sites.
  • Training and Certification Programs - to educate staff, through courses and other means, on the conduct of high quality clinical research including training in "Good Clinical Practices." Some training courses, such as Certified Clinical Research Coordinator (CCRC) courses, provide certification. Taking the on-line NIH training course in human subject protections and passing the on-line examination are mandatory for all individuals participating in NIH clinical research projects. Training on the center's SOPs and study processes is also important. Documentation of training activities that includes the type of training and the data completed should be stored in study files or binders.

3.0. Basic Study Elements to Enhance Study Quality

The general purpose of a research project is to collect data that enables the investigators to test the study hypothesis and achieve the study objectives. The data collection and data management processes are as important in achieving the study objectives as having a scientifically sound study design and protocol.

To assist in preventing problems with data quality, investigators must incorporate standard research methodology into their studies. It is essential that study investigators and clinical centers follow and adhere to an approved, common protocol to assure that data collected can be aggregated for analysis and interpretation.

Key components of a protocol are the same for single and - or multi-center clinical studies. The protocol provides a statement of the problem, study objectives, research questions and hypotheses. The NINDS has developed a template for protocols (available in a Word format).

Training study staff at clinical centers helps to ensure that the protocol is applied in a standardized way across sites and participants by all study staff. Formal training sessions for investigators, site coordinators, and other staff as necessary, are held to review the study goals and procedures, study forms, and often include a certification process.

Accepted research methods and guidance on clinical trials help to ensure that the data required addresses that the study questions are collected and recorded in the same way across sites, investigators, and participants. These methods and approaches are outlined and briefly described below.

3.1 Study Administration and Manual of Procedures (MOP)

The clinical study project team is responsible for developing study materials; recruiting, screening, enrolling, and following study participants; collecting study data, ensuring data quality, and analyzing and presenting study results. A project organization chart identifies each staff member's responsibilities and communication lines within the center and across sites in multi-center studies. Multi-center studies may involve a clinical coordinating center and data coordinating center, central laboratories and reading centers, drug distribution centers, and numerous clinical sites. Written job descriptions that describe responsibilities and backup staff are important for staff members within the clinical sites and coordinating centers.

The Manual of Procedures (MOP) is the document that transforms the protocol into an operational research project. It details the study organization, the definitions of data elements to be collected, visit procedures, data management, data flow, case report forms (CRFs), safety monitoring, and quality control procedures. It documents the study flow so that the screening, initial evaluation, enrollment, baseline evaluations, randomization, treatment and follow-up of all study participants are conducted in a structured and standardized manner. It details how the data are observed, collected, and recorded. The MOP should include sufficient detail so that it could be used as a training manual for new study staff. (Please see Guidelines for Developing a Manual of Procedures (MOP)(pdf, 479 KB) (PDF, 468kb).

Before a study begins recruitment, a study flow should be developed that details each contact with study participants or potential study participants, time period, and data that must be collected. Other useful mechanisms to prevent issues with data quality include participant files and a study administrative file or the study binder.

The participant files may contain the participant's informed consent form, source documents, such as copies of the medical record (because these are often difficult to retrieve in a hospital setting), laboratory results, MRI scans, and other data related to the study, telephone notes, videos, and instruments or tests completed by the participant.

The study binder contains all protocol documents and amendments, investigator brochures, staff C.V.s, financial disclosure forms to document that there is no conflict of interest, CRF copies, copies of advertisements used by the site to recruit participants (IRB-approved advertisements), IRB approvals, informed consent forms, important study communications from the IRB, NIH, and, if relevant, the FDA and/or coordinating center, laboratory certifications, monitoring reports, and any other information given to study participants, participant identification codes, and copies of serious adverse event reports.

3.1.1 Conflict of Interest

NIH guidelines specify that investigators should not have a financial interest in a study intervention and that they must comply with 42 CFR Part 50 Subpart F and/or CFR Part 94 as well as the Institution's policy.

Investigators must not have any intellectual or scientific conflict of interest with the conduct of the study. Further, each local institution has policies and procedures in place that all investigators must comply with, and which govern any potential conflicts of interest. Procedures should be in place to help investigators and other key personnel comply with all requirements.

3.1.2 Drug/Device Procedures

Written procedures document how the study treatment, if relevant, is to be stored, prepared, dispensed, and returned. Procedures must also describe instructions for completing drug accountability and administration records.

"Pharmacy" refers to the unit responsible for the storage, dispensing, and accountability for the investigational agent. An actual pharmacy may or may not be directly involved in a study at the clinic level since the investigational agent may be delivered directly to the clinic site in pre-labeled sealed packages.

3.1.3 Participant Materials/Samples

Written procedures document how laboratory samples, MRIs, and other study participant samples, documents, or materials are to be collected, labeled, handled, shipped to the coordinating center, and tracked so that study data are not lost. As stated in the new Health Insurance Portability and Accountability Act (HIPAA) guidelines, personal identifiers such as name, geographic location, social security number, and fifteen other specific individual identifiers should not be used. Thus when transmitting participant materials it is important to specify how the material is to be identified (e.g. the participant identification number).

3.1.4 Recruitment and Retention Plan

A common issue that can impact the study timeline and study results is participant recruitment. A comprehensive recruitment plan helps to ensure that a study will enroll the target number of participants in a timely manner. Study staff must define the characteristics and needs of the target population. Once the features and needs are defined, the staff must identify appropriate channels for communicating study information. The recruitment plan details strategies, for instance, through community physician referrals, clinical site chart reviews, and/or radio, television or printed advertisements. The plan may specify, for example, that the Principal Investigator will host dinners for local physicians every three months.

Participant retention also requires planning to meet the needs of the target population. A common issue that can threaten the validity of the study results is a low retention rate.  If participants are lost to follow up, their data is still included in the analysis under the intention to treat principle.  If no observed data are available, then data are imputed which may decrease the validity of the results.  For example, if the target population includes patients with early multiple sclerosis (MS), holding clinic hours before and after work may aid in participant retention.

3.2 Pre-Screening and Screening

For every enrolled study participant, sites typically identify and screen many potential participants. To assist with identifying potential participants, sites review clinic or admission logs through a pre-screening process. Information captured during the pre-screening process is usually documented in a log that captures initials, gender, age, and other minimal information obtained from a clinic or telephone call. The log documents reasons for ineligibility and for non-enrollment of eligible participants. It is generally maintained locally and may be requested by the coordinating center.

3.2.1 Screening

If an individual appears to be potentially eligible for a study, they are asked whether they may be interested in study participation. In adherence to 45 CFR 46.116 and 21 CFR 50.20, an individual must sign an informed consent form prior to having any study specific tests or  screening procedures. Screening or pre-enrollment study forms correspond to screening procedures so that eligibility and ineligibility criteria are efficiently captured.

As soon as an individual is determined through the screening process to be ineligible, the screening process should terminate, so that the person is not enrolled in the study in error. If ineligible participants are enrolled in the study, and a few usually are, there should be procedures in place to document these and other protocol violations. Enrolling participants that do not meet the study inclusion criteria and meet one or more of the exclusion criteria is an issue that frequently compromises study quality.

3.2.2 Informed Consent

The informed consent process and participant safety, in general, have come under increased scrutiny recently by the DHHS Office for Human Research Protections (OHRP). The informed consent form contents, informed consent process, and use of current, IRB approved versions of the informed consent form are all important aspects of patient safety. Written procedures should document at what time point in the study flow the participant must sign the correct version of the informed consent form. The NIH Office of Science Policy in collaboration with the Office of Extramural Research (OER) and the Extramural Affairs Working Group (EAWG) published a new resource, Informed Consent for Secondary Research with Data and Biospecimens. This resource provides points to consider and sample language for informed consent documents of research studies which plan to store and share data and/or biospecimens for future use. If you have questions or need assistance developing your consent language, please feel free to contact NINDS Clinical Research Liaison (any consents).

Informed consent issues are some of the most common study quality breaches that are found during study audits.

3.2.3 Randomization and Blinding

Randomization is the method for assigning participants to treatment groups to ensure that their characteristics are evenly distributed among these groups. In many randomized studies, all participants and/or treating physicians are "masked" or "blinded" to the treatment assignment and do not know which intervention is administered.

In clinical trials that employ randomization, blinding is strongly encouraged. In double-blind studies, participants and the treating clinician are "blinded" or "masked" to the treatment and do not know if the participant is receiving drug or placebo. In surgery studies and other studies in which it may be difficult to blind the treating physician, typically the clinician who assesses the patient at baseline and during follow-up visits remains blinded.

The study statistician generates the randomization numbers and/or a designated study staff member keeps the randomization assignments under "lock and key" to protect the identification of the study treatment assignments. Generating more randomization assignment numbers than will be needed is recommended. The format for assigning treatment groups through randomization should prevent the investigators from accurately guessing to which treatment group a patient will be assigned.

The statistician or data management group details the procedures that sites must follow to randomize a participant and ensures that randomization is applied consecutively. Randomization and blinding/unblinding procedures should be determined prior to the enrollment of the first participant.

Unmasking/unblinding is a serious action that should be restricted in order to limit potential bias. The Data Safety Monitoring Board (DSMB) or Safety Officer and the NINDS Program Official must be involved in the decision to unmask/unblind and must grant approval for unmasking/unblinding any individual. However, if a participant is in a life-threatening situation in which knowledge of the treatment is crucial to care, the investigator may unmask the participant without prior DSMB or NINDS approval and must then immediately notify the study statistician and submit a detailed report detailing the unmasking event. If unmasking/unblinding occurs, the statistician should record and maintain the following in a confidential log:

  • ID number of the participant whose treatment assignment was unblinded/unmasked;
  • Date;
  • Reason for unmasking/unblinding study staff;
  • Person responsible for unmasking/unblinding study staff; and
  • List of persons who are unmasked/unblinded.

3.3 Study Forms and Data Collection

Study forms, often called Case Report Forms (CRFs), are used to collect and document all data and study observations in a standardized manner at specific time points over the course of a study. Forms are organized by study visit to assist with data collection and should ultimately correspond with data analysis plans. Screening and enrollment forms contain data that document inclusion and exclusion criteria, such as demographic profile, medical history, vital signs and physical examination. Typical data collection time points, in addition to screening and enrollment, include baseline examination, treatment interventions, follow-up, and study completion. A form documents each participant's end of study status, whether it be "completed according to protocol," terminated, discontinued, died, or "lost to follow-up." Forms also document adverse events. Study forms should not include any information that may identify study participants. A schedule of participant contacts lists study visits along with tests, examinations and questionnaires, and helps to plan the study data collection forms.

Data that are collected for a study should be limited to those elements that will answer the study questions. Some studies attempt to collect hundreds of pages of data elements, many of which are nice to have, but are not necessary to answer the study hypotheses. Pre-testing study forms on several participants before a study commences allows the study staff to identify and correct problems, including content or format. Study forms should not include any information that might personally identify study participants.

Data collection procedures should define how each data element will be collected at each participant contact. The study statistician can assure that the data that will be collected and the study forms will address the analysis plan. Study forms must capture necessary adverse event data to generate safety reports.

It is the responsibility of the site staff to ensure that completed forms are accurate and complete. It is important for the study staff to follow study participants even though the study treatment may be discontinued.

Study data collection procedures do not substitute for patient care documentation in the medical record. Maintaining hard copies of medical records, including hospital or clinic records, study-specific documents, telephone notes, laboratory records, videos, and instruments or tests completed by the participant, will provide the documentation necessary to assure that all required data are reported accurately.

3.4 Data Management

Clinical studies are data intensive efforts. Procedures should describe the flow of data from the data collection process through analysis. Data management staff members should develop a data management plan that documents the data flow within the clinical sites, transfers the data from clinical sites, handles error identification and resolution identifies useful reports, including steps to derive an analytic database from edited or "clean" records. To implement a system that meets study and regulatory requirements, investigators should involve information technology (IT) staff and colleagues with clinical data management expertise and experience, especially in multi-center studies. Procedures should also document how participants and forms flow through a study so reports that describe enrollment, study status and forms status can be generated. As relevant, procedures should also describe data cleaning and correcting as well as data handling for analysis. Data checks for ranges, cross-validity and completion are done proximal to the data collection.  Automated checks at data entry may be helpful.

Study computer systems and analytic programs also require documentation, validation, and verification of data to ensure their accuracy and integrity.

3.5 Safety Monitoring and Human Subjects Protection

NIH policy requires that the clinical research projects it sponsors have a system in place for independent oversight and monitoring of the study, to ensure the safety of participants and the validity of the data (NIH Policy for Data Safety and Monitoring, NIH Guide, June 12, 1998).

The NINDS requires that investigators supported by the Institute implement data and safety monitoring procedures, regardless of the phase of the study (NINDS Guidelines for Data and Safety Monitoring in Clinical Trials).

The safety monitoring plan is reviewed as part of the peer review process and must be detailed in the protocol document. Specific monitoring requirements depend on the study type. Multi-center and high risk studies have a Medical Safety Monitor that reviews Serious Adverse Events (SAE) reports in a "real time," blinded manner to quickly identify potential safety concerns. If concerns arise, the Medical Safety Monitor notifies the NINDS Program Official and/or the Data and Safety Monitoring Board. The study's safety monitoring procedures describe what is reported to whom in what time frame.

4.0. Quality Assurance

4.1 Quality Control Plan

While computer systems can collect, edit, store, and report data, the investigators, data management specialists, and statisticians must identify and routinely prepare reports that describe study progress and issues. The reports can alert the investigators to potential problems before they compromise the study. Thus, identification of reports and procedures for ongoing data review and site monitoring form a quality control plan.

Documentation and plans for review of study systems and analysis programs are also part of a study's quality control plan. This review should be carried out before the software is used.

4.2 Detecting Quality Assurance Issues During a Study

Once a study begins, investigators can assure the highest quality data by following procedures established to review reports, monitor sites, and check data.

Ongoing review of reports that describe study operations and present data by site help to ensure that problems are identified and corrected in a timely manner. No study is perfect and thus, a critical review early in a study may help identify problems that can be resolved without compromising results.

The following reports are often generated for routine review:

  • Screening, recruitment, enrollment, and retention reports - illuminate recruitment issues and problem sites
  • Protocol compliance reports - identify protocol violations
  • Data quality reports - describe missing, erroneous, and inconsistent data to ensure protocol is followed and deviations are tracked.
  • Identification/reporting of serious adverse events - to ensure that adverse events are identified and reported in a timely, comprehensive and accurate manner
  • Site monitoring and data reports - provide feedback on issues discovered during site visits and verify validity and completeness of data

Systematic visits to clinical sites that include review of study procedures and data help to ensure that the protocol is followed. After a visit, the site visitors prepare a report for the investigators and NINDS. The report documents the visit and provides site staff with a review of study progress and with comments and suggestions for improving the conduct of the study.

4.3 Setting and Maintaining Quality Standards

One approach to detecting quality and integrity issues and improving performance during a study is to establish standards. Examples of standards to measure quality include the following:

  • Study Participants - Enrolled participants meet all inclusion and exclusion criteria and no more than a percentage of participants (e.g. 5%) do not meet these criteria at a site.
  • Enrollment - Study enrollment goal is achieved (at least 90%) on time.
  • Drop Outs - Study drop-out rate is not greater than 5%.
  • Data Entry - Error rate is not greater than 0.001%.
  • Analysis - Data are analyzed using standard, generally accepted statistical software.

Rates exceeding pre-specified percentages trigger a correction process. Likewise, if standards are not met, a correction or training process can be implemented.

4.4 Quality Correction Procedures

If data quality and integrity issues are uncovered during a study, there must be processes in place for systematic correction. Examples of correction processes include the following:

  • Data - Changes to forms are documented in ink with a single line drawn through the incorrect item and the correct response noted, along with the date and the initials of the person making the correction. For data stored in a computer, there should be an audit trail of all changes.
  • Systematic Errors - Systematic errors may result in changes to the protocol and MOP. For example, in a study of participants with dementia, investigators may find that some participants cannot complete a battery of tests. However, "Not done" is not an allowable choice on the forms. Thus, the form and computer system may need to be changed.
  • Detection Processes - Reports and procedures may require modification. For example, if recruitment is slow in a study, the Coordinating Center may request that the sites fax or email their screening logs. A new report may be developed that captures screen failures across sites to better understand if there is a need to change study procedures.
  • Protocol Violations - If there appear to be issues symptomatic of one or a few sites, retraining may correct the problem. For example, although the protocol states that randomization must occur within three days of the screening visit, some sites are not adhering to this rule. Retraining site staff may be an appropriate initial step.

5.0. Summary

The Guidelines provide a context for prevention, detection, and correction of clinical study data quality and integrity issues. Investigators can tailor the suggestions to meet the requirements of a study and can expand upon them as they develop and expand their own quality assurance methods.