How do I review thee? Let me count the ways: a comparison of research grant proposal review criteria across US federal funding agencies.

Author:Falk-Krzesinski, Holly J.
 
FREE EXCERPT

Introduction

The research funding landscape in the United States is highly competitive, with flat or shrinking budgets for investigator-initiated research programs at most federal agencies (American Association for the Advancement of Science (AAAS), 2014). Taking biomedical research as an example, in 2014, the National Institutes of Health (NIH) budgeted $15 billion to fund research project grants, an amount that has essentially remained the same since 2003 (AAAS, 2014; Federation of American Societies for Experimental Biology, 2014). At the same time, the number of research grant applications has steadily increased, from close to 35,000 in 2003 to 51,000 in 2014. The result has been a stunning 30% drop in funding success rates, from 30.2% in 2003 to 18.8% in 2014. Other federal agencies that fund research, including the National Science Foundation (NSF), Office of Veterans Affairs (VA), and Department of Defense (DoD), are feeling the similar sting of budget restrictions.

Within this tenuous funding environment, it has become essential that investigators and research development offices sustain their research programs by continuing to encourage new researchers to apply for grant support and encouraging established researchers to diversify their funding portfolios. New researchers benefit from clear information about the federal grant process, and experienced researchers benefit from considering funding opportunities from federal funding agencies, national organizations and advocacy groups, state agencies, private philanthropic organizations, regional or local special interest groups, corporations, and internal institutional grant competitions that may not be their typical targets for support. With increasing competition for grant funding, investigators who might be accustomed to one set of rules for preparing grant proposals may become quickly overwhelmed by the prospect of learning entirely new sets of rules for different funding agencies.

Yet this process is not as daunting ifwe start from the perspective that any funder that offers research grants has essentially the same goal: to support research that fits within its mission and will bring a strong return on its financial investment (Russell & Morrison, 2015). The review criteria used to evaluate research grant proposals reflect the funder's approach to identifying the most relevant and impactful research to support (Geever, 2012; Gerin & Kapelewski, 2010; Kiritz, 2007). Thus, planning and preparing a successful grant proposal depends on a clear understanding of the review criteria that will be used. These criteria directly inform how the proposal content should be presented and how much space should be afforded to each section of the proposal, as well as which keywords should be highlighted. It may seem that each funder--federal, state, local, private--has its own distinct set of rules regarding the preparation and review of grant proposals, and that each funder uses specific jargon in its review process. However, because all funders aim to support research that is relevant and impactful, we suggest that the mandatory review criteria used to evaluate research grant proposals are based on a set offundamental questions, such as: Does this research fit within the funder's mission? Will the results of this research fill a gap in knowledge or meet an unmet need? Do the investigators have the skills and resources necessary to carry out the research?

In this article, we examine the research grant proposal review criteria used by 10 US federal agencies to demonstrate that there exist only a small and finite number of ways that federal research grant proposals are actually evaluated. Our goal is to help research administrators and research development professionals empower investigators to more confidently navigate funder review criteria, thereby lowering the barrier to first-time applicants or to grant portfolio diversification for more established researchers. Recognizing that research proposal review criteria are aligned across federal funding agencies can also help proposal writers who might be faced with other funding opportunities in which the review criteria are not clearly defined. On the flip side of that equation, understanding that review criteria are based on the same core goals can help grantmakers as they develop and refine review criteria for their funding opportunities.

Observations

We performed an online search of 10 US federal agencies' (NIH, NSF, VA, Department of Education [ED], DoD, National Aeronautics and Space Administration [NASA], Department of Energy [DOE], United States Department of Agriculture [USDA], National Endowment for the Humanities [NEH], and National Endowment for the Arts [NEA]) websites to identify policies and procedures related to their research grant proposal review process. The NIH Office of Extramural research (OER) website provided the greatest detail and transparency with regard to the review criteria and review process used for evaluating research grant proposals (National Institutes of Health, 2008a; 2008b; 2015 a), and served as a starting point for our analysis of the review criteria for the other nine agencies. We developed key questions corresponding to each of the NIH review criteria, and then aligned the review criteria of the remaining nine agencies with these key questions.

Federal grant program guidance and policy changes occur frequently; the links to online resources for research grant proposal policies for each of the various funding agencies included in our analysis were current as of August 10, 2015. Note that our analysis includes information from the National Institute on Disability and Rehabilitation Research (NIDRR) program as administered by ED. On June 1, 2015, the NIDRR was transferred from ED to the Administration for Community Living (ACL) in the US Department of Health and Human Services (DHHS), and is now called the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) Field-Initiated Program. Our analysis of NIDRR was current as of May 4, 2015.

Also note that there is variability between different research grant programs within each federal agency. We included in our analysis review criteria from the DoD Congressionally Directed Medical Research Programs (CDMRP), the USDA National Institute of Food and Agriculture, the NEH Digital Humanities Start-up program, and the NEA ART WORKS program. Criteria for NASA research programs were compiled from numerous NASA Research Announcements.

The NIH review criteria

The NIH criteria emphasize clinical, interdisciplinary, and translational biomedical research (National Institutes of Health, 2008a). Reviewers are instructed to evaluate research grant proposals based on how well five core review criteria are met: Significance, Innovation, Approach, Investigator(s), and Environment (Table 1) (National Institutes of Health, 2015a; 2015b). Assigned reviewers consider each of the five core review criteria and assign a separate score for each using a 9-point scale. These ratings are included in a summary statement that is provided to the researcher, whether or not the entire study section ultimately discusses the proposal.

Each of the five core review criteria can be simplified into a general question. The Significance criterion asks reviewers to consider "Why does the research matter ?" Reviewers look for whether the proposed project will address an important problem or critical barrier to progress in the field, and whether the knowledge gained from the proposed research will advance scientific knowledge, technical capacity, or clinical practice to drive the field forward. Innovation translates into "How is the research new?" Reviewers consider how the proposed research challenges current thinking with novel concepts, approaches, tools, or treatments. Approach asks, "How will the research be done ?" Reviewers assess the proposed research strategy, methodology, and analyses and determine whether they are appropriate to achieve the aims of the project, and how riskier aspects of the proposal might be handled with alternative approaches. The remaining two core criteria evaluate the context in which the research will be done--defined as the collective set of resources, equipment, institutional support, and facilities available (Environment)--and what is special about the people doing the research (Investigator). For the Environment criterion, reviewers evaluate whether the resources and institutional...

To continue reading

FREE SIGN UP