Understanding Paylines

Paylines and Percentile Scores: Understanding the Difference

In order to understand how NINDS makes funding decisions, it is first important to understand the concept of paylines and percentile scores. 

Percentile Scores 

Let’s start with percentile scores. Percentiles indicate relative rank. For unsolicited applications reviewed by the Center for Scientific Review (CSR), NIH converts an overall priority score into a percentile. A percentile ranks an application relative to the other applications reviewed by a study section at its last three meetings. Percentiles range from 1 to 99 in whole numbers. In contrast to usual mathematical practice, a lower number indicates a better score. 

NIH uses percentiles to counter a phenomenon called "score creep" where study sections give applications increasingly better scores. As a result, scores cluster in the exceptional range, making it impossible to discriminate among applications. Each study section can apply the NIH review criteria differently, scoring either more harshly or more favorably. Percentiling counters these trends by ranking applications relative to others scored by the same study section. Altogether, percentiling makes review fairer to applicants. Our colleagues at NIAID go into further detail, describing how percentiles are calculated using sample data.

Payline 

Next, we’ll cover how a percentile score is related to the Institute’s payline. Each Fiscal Year (FY), the Institute identifies a percentile score that serves as “payline”, or funding cutoff, beyond which most applications will not be funded. NINDS sets paylines for R01, R03, R15, and R21 applications by balancing the amount of funds available to spend on new grant awards with an estimated number of applications we anticipate will score well based on historic trends. More details about how the Institute establishes its payline can be found below. 

Recognizing the diversity of the grant portfolio, NINDS views paylines as the fairest way to make most funding decisions. A numerical value lets the Institute cut across disciplines and fund the best science as determined by peer review. It is important to remember that paylines vary among NIH institutes, so a percentile or overall priority score that is not fundable in one institute may be fundable in another. 

 

 

 

Image

How NINDS sets paylines 

We’ve talked about what a payline is and how it’s used at NINDS.  But the process of determining what the payline will be in a given year is quite complex.  Below we describe some of the considerations that NINDS (and most NIH ICs) must account for when setting paylines.  As we’ve discussed elsewhere, the Federal appropriations process is a complex one, and often NIH doesn’t receive a final budget until mid-way through the fiscal year. Using all the information and historic data we have at our disposal, NINDS sets a conservative payline at the beginning of the fiscal year, with the hope of being able to raise it after a budget is finalized. However, even knowing how much money is available to spend in a given year isn’t quite as simple as it seems. Beyond that, many other factors go into setting the payline, including:  

• Money tied up in “outyear commitments”. Most NIH grant awards are made for more than 1 year; however, when NIH awards a grant, it pays out one year at a time.  If you receive a 5-year R01, you typically receive 1 year of funding in the first year and additional installments in each of the remaining 4 years. NIH “commits” to paying the additional four years of money; however these funds come out of each subsequent appropriation.   

• Funds for new and established programs. At any point in time, the Institute commits money to support programs that capitalize on scientific opportunities, build capacity in needed areas of the scientific workforce, and/or catalyze understudied research areas.  Based on the published “set aside”, funds are held in reserve to support meritorious applications submitted in response to these Notices of Funding Opportunities. It is important to remember that the large Congressionally-supported programs (BRAIN, the AD/ADRD program, HEAL, and the SBIR program) receive funds specifically to support them, and thus do not have any bearing on the NINDS payline. 

• Trends in grant applications. It is difficult to predict how many applications NINDS will receive and how much they will cost at any given payline point, but this is one of the main factors the Institute grapples with each year.  This has become increasingly complex, as we have reached an inflection point in the number and size (in dollars) of applications we receive. Grant applications are at historic highs, while the number of PIs that NIH supports is stable, relatively speaking (see Research Project Grant Investigators: Number of Competing Investigators and Funding Rates chart). Moreover, increasing grant budgets make this calculus even more challenging.