Determinants of broader impacts activities: a survey of NSF-funded investigators.

Author:Nagy, Dianne

Background and Objectives

An enduring image of academia is that of an ivory tower, disconnected from the messy problems of the world. It is a recurring complaint that the primacy of basic research and the academy's emphasis on abstract theory has eroded higher educations connection to the world, isolating scholars from society and making their work obscure or irrelevant to the general public (Barker & Brown, 2009: Gibson, 2006). Decreasing appropriations for higher education heighten the need to convince the public about the value of university research. In response to prodding from Congress, federal funding agencies are increasingly requiring academic research grant proposals to include indicators of public impact that would result in social good.

In 1997, the National Science Foundation (NSF) established a policy that all funding proposals submitted to the agency would be evaluated on two criteria: intellectual merit and broader impacts. Broader impacts refer to "specific, desired societal outcomes" (NSF, 2012, p. III-2) such as the participation of underrepresented groups in science, technology, engineering, and mathematics (STEM); enhancing STEM education; public scientific literacy and engagement; and partnerships between academia, industry, and others. According to Arden Bement, former director of the NSF, "The [broader impacts] criterion was established to get scientists out of their ivory towers and connect them to society" (quoted in Lok, 2010, p. 416).

A national review of the effectiveness of these criteria conducted in 2000 revealed that, while most researchers had little difficulty identifying the intellectual merit of their plans, many struggled to adequately articulate the broader impacts of their proposed investigations (NSF, 2005). The study also found that the broader impacts criterion was consistently weighted less than intellectual merit in the proposal review process and that many in the scientific community were resistant to or dismissive of the requirement (NSF, 2005). The NSF responded with efforts to educate the scientific community about the agency's rationale and expectations for broader impacts. A subsequent national evaluation of the NSF's review criteria conducted in 2010 found that problems with the execution, understanding, and acceptance of the broader impacts criterion persisted, that assessment was unclear and inconsistent, that there was little variety in the type of activities performed to address the broader impacts criterion, and that principal investigators (Pis) needed greater institutional support to respond effectively to this requirement (NSB, 2011).

The purpose of this study was to investigate the factors that shape PI response to the NSF broader impacts criterion. The following research questions guided the study:

  1. What are the different types of activity that PIs engage in to meet the broader impacts criterion of NSF-sponsored research?

  2. What is the quality of the broader impacts activities, as assessed by PIs using evaluation criteria for the scholarship of engagement?

  3. To what extent do PIs' individual characteristics (expertise, epistemology, academic discipline, and rank) and institutional support (climate and resources) for community engagement predict the type of their broader impacts activities?

  4. To what extent do PIs' individual characteristics and institutional support predict the quality of their broader impacts activities?

The author gathered survey data to identify the types of broader impacts activities that Pis conduct, and their perceptions of the quality of these activities. The author then determined how much variance in type and quality of activity could be explained by the respondents' personal characteristics and perceptions of institutional support for community engagement. In the survey, community engagement was defined as "The collaboration between institutions of higher education and their larger communities (local, regional/state, national, global) for the mutually beneficial exchange of knowledge and resources in a context of partnership and reciprocity" (Carnegie Foundation for the Advancement of Teaching, n.d., n.p.).

Conceptual Framework

This study is grounded in the theoretical framework of the scholarship of engagement. Such scholarship is academically relevant faculty work that simultaneously meets campus goals and community needs, incorporates community issues, and is integrative across teaching, research, and service (Clearinghouse and National Review Board for the Scholarship of Engagement, 2002). Although there are subtle distinctions between the

terms "scholarship of engagement," "engaged scholarship," and "community-engaged research," they are used interchangeably throughout this study.

The NSF's broader impacts merit review criterion has been an ongoing challenge within the scientific community, particularly in terms of conceptual clarity, assessment, and philosophical resistance. The persistence of the problem may stem, in part, from the paucity of literature on the broader impacts of academic research (Buxton, 2011). While scant research has been conducted on such impacts, volumes have been written on the scholarship of engagement. The engagement movement has spawned multiple journals, institutes, conferences, and consortia dedicated to studying engaged scholarship in its many forms. This study drew from this extensive literature base to better understand the issues that surround the broader impacts of academic research and connections between the academy and society. Framing the study within the context of the scholarship of engagement is appropriate not only because broader impacts activities may be viewed as an expression of engaged scholarship, but also because both struggle to achieve legitimacy and support within the academy.

The Scholarship of Engagement

The concept of engaged scholarship is often credited to Ernest Boyer, a renowned leader of educational reform. Boyer (1990) argued that the traditional definition of scholarship as research that advances disciplinary frontiers of knowledge was too limited. He countered the prevailing hierarchical view of scholarship with a more inclusive vision that added to the ranks of conducting original research such activities as identifying connections between concepts, bridging theory and practice, and effectively communicating knowledge. Boyer posited that faculty scholarly work included four "separate, yet overlapping, functions" (1990, p. 16) which he identified as the scholarships of discovery, integration, application, and teaching. Boyer (1996) proposed the term scholarship of engagement as interaction across these four realms to address community needs. He described the scholarship of engagement as "connecting the rich resources of the university to the most pressing social, civic, and ethical problems ... creating a special climate in which the academic and civic cultures communicate more continuously and more creatively with each other" (pp. 1920). Boyer asserted that it was time for higher education to renew its covenant with society and to partner with communities in directly addressing problems, with a reciprocal flow of knowledge.

Models of Knowledge Production

Epistemologies about knowledge production and distribution shape perceptions about the role of higher education in society, define relationships between campus and community, and govern the values, norms, and practices of scholars. Boyer raised critical questions of how knowledge is constructed and what is accepted as legitimate knowledge in the academy. Deeply embedded in academia is the traditional linear model of knowledge production and distribution, which prizes basic research that is disciplinary, removed from influence by outside interests, and conducted without consideration of use (Bush, 1945). Boyer countered the prevailing belief that basic research was the most essential form of scholarly activity, with publications and teaching flowing from it. He argued that "knowledge is not necessarily developed in such a linear manner. The arrow of causality can, and frequently does, point in both directions. Theory surely leads to practice. But practice also leads to theory" (1990, p. 16). The national dialogue that ensued prompted academics to identify new models of knowledge creation, including that of use-inspired research (Stokes, 1997) and Mode 2 science (Gibbons et ah, 1994).


The dominant, linear model of knowledge production begins with basic research, which may lead to applied research and technological development, and on to production or operations. This model has shaped U.S. science policy since it was first proposed by Vannevar Bush in his 1945 report to President Roosevelt (Mazuzan, 1994). Bush saw an inherent tension between the goals of understanding and use, and warned that the creativity of basic research would be stifled by premature thoughts of practical use.


One year after Boyer's call for engaged scholarship, Stokes challenged the traditional dichotomy between basic and applied research. Stokes (1997) argued that Bush's premise that flows between science and technology are uniformly one way, from scientific discovery to technological innovation, was flawed and offered a limited understanding of how knowledge is generated and put to use. Stokes contested Bush's cannon that the goals of understanding and use are dichotomous, arguing that research is often influenced by both goals. In his model, theoretical and practical research and application come together to create a dynamic cycle of innovation driven by changing conditions and the competitive landscape.

Mode 2

Gibbons and his co-authors (1994) offered another alternative to the static linear model of knowledge production. They determined that Bush's old paradigm of discipline-specific, autonomous, university-grounded scientific discovery ("Mode 1") was being supplanted by a new approach to...

To continue reading