Back to Basics

• Published date: 01 June 2017
• Author: Luis Ignacio Alvarez‐Gonzalez,Marta Rey‐Garcia,Karen Maas,Kellie Liket
• DOI: http://doi.org/10.1002/nml.21259
• Date: 01 June 2017

493

N M  L, vol. 27, no. 4, Summer 2017 © 2017 Wiley Periodicals, Inc.

Published online in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/nml.21259

Journal sponsored by the Jack, Joseph and Morton Mandel School of Applied Social Sciences, Case Western Reserve University.

Back to Basics

REVISITING THE RELEVANCE OF BENEFICIARIES

FOR EVALUATION AND ACCOUNTABILITY IN NONPROFITS

Marta Rey-Garcia , 1 Kellie Liket , 2 Luis Ignacio Alvarez-Gonzalez , 3

Karen Maas 2

1 University of A Coruña , 2 Erasmus University Rotterdam , 3 University of Oviedo

Nonprofits are under increased accountability pressures to demonstrate their effectiveness.

Output measurement (how much is produced) is disregarded as simplistic. Emphasis is

made instead on measuring outcomes (changes in the lives of beneficiaries) or impacts

(effects developed relative to the mission of the nonprofit, or the overall public good),

and a growing portion of organizations state that they measure these effects. However,

we question the assumption that outputs such as the number of beneficiaries served are

being adequately measured. We first review existing research gaps on results measurement

practices and discuss the main types of obstacles to the quality and utility of evaluation

data. In this context, we argue for the need to reground nonprofit evaluation in the

profound knowledge available about beneficiary populations. We discuss the potential

and limitations of reach, a basic output indicator that is defined as the number of

individuals directly affected by a nonprofit, and explore the organizational drivers of

reach measurement. Evidence from 2,229 nonprofits shows they still lack adequate data

on the beneficiaries they serve, face relevant conceptual and practical hurdles when try-

ing to identify them, and are significantly influenced by organizational factors in their

capacity to track them. Our research not only shows that nonprofits fail to adequately

measure outputs, but also that measuring the number of beneficiaries served and how

they are served is not as straightforward as outcome and impact advocates suggest. Prac-

titioners and funders are reminded of the need to place beneficiaries at the core of their

evaluation efforts.

Keywords: nonprofits , beneficiaries , impact evaluation , accountability , reach

Correspondence to: Marta Rey-Garcia, University of A Corun

~a, School of Economics and Business, A Corun

~a 15071,

Spain. Email: martarey@udc.es.

Authors thank the late Woods Bowman , Mark Hager, and three reviewers for their valuable comments to earlier versions of this article.

MR-G and LIA-G acknowledge funding from the Spanish Ministry of Economy and Competitiveness under the research project entitled

“Marketing and Social Innovation” (ECO2013-46391-R). Furthermore, MR-G thanks Prof. Royo for the opportunity to work on this

article as visiting scholar to Suff olk University, Boston.

Nonprofi t Management & Leadership DOI: 10.1002/nml

494 REYGARCIA, LIKET, ALVAREZGONZALEZ, MAAS

ACCOUNTABILITY ENTAILS INVITING the public to scrutinize the behaviors of organizations’

leadership (Hoefer 2000 ). Nonprofits are particularly susceptible to scrutiny because of their

mission oriented toward achieving socially valued goals, the complexity of societal demands

and expectations around them, and fierce competition for paid and voluntary resources. These

pressures have increased over the past two decades because of a combination of social, ethical,

and regulatory forces (Benjamin 2013 ; Carnochan et al. 2014 ), and because accountability is

now considered to be “essential for the effective regulation and monitoring of nonprofits by

beneficiaries, donors and governments” (Burger and Owens 2010 , 1263). Accountable behav-

iors generate trust among relevant stakeholders and help nonprofits build the material and

intangible resources they need for long-term survival and legitimacy in the eyes of policymak-

ers and society (Fernandez 2008 ).

This movement started with demands for increased transparency, but in the past decade has

focused on demonstration of the difference organizations make in tackling social problems.

In response, numerous evaluation tools, frameworks, and approaches have been developed.

It has been rightly argued that evaluations should focus on outcome- and impact-level effects

to measure the change that nonprofit programs make in the lives of the individuals who are

their beneficiaries (Benjamin 2012 , 2013 ; Ebrahim and Rangan 2010 ; Eckerd and Moulton

2011 ; MacIndoe and Barman 2012 ). However, nonprofits tend to measure “things that are

‘easy’ to measure, such as the number of clients served, quantity of program units delivered,

activities provided, or number of volunteer hours contributed” (LeRoux and Wright 2010 ,

574). In this context, many practical guides have included statements such as “simply count-

ing people served … won ’ t cut in today ’ s environment” and “[some grantees] report the num-

bers of participants they reach … as though they were results” (Benjamin 2012 , 440).

But are we right to assume that nonprofits adequately measure basic outputs such as the

number of people served? In this article we first review literature on the extent of measure-

ment practices and existing barriers to meaningful engagement with evaluation data. In this

context, we discuss why reach —defined as the number of individuals who are the direct

beneficiaries of nonprofit interventions—might be a stepping-stone for substantiating larger

evaluation and accountability efforts, and explore the organizational factors that influence

the capacity of nonprofits to measure it. Based on evidence from 2,229 nonprofits, the fol-

lowing questions are addressed: Are nonprofits actually measuring their reach? What are the

practical barriers to reach measurement? Which nonprofits measure their reach? Conclusions

and implications follow.

Are Nonprofi ts Actually Measuring Their Results?

Practical Barriers to Meaningful Evaluation

Limitations of Existing Studies on Measurement Practices

Until recently, the “big picture” on the status of evaluation in nonprofits was largely unknown

because of a “serious shortage of rigorous systematic evidence” (Henry and Mark 2003 , 69).

In the past decade, numerous efforts have been made to gain more insight into these evalu-

ation practices. However, our knowledge of the extent to which outcomes and impact-level

effects are measured, and of the quality and utility of evaluations, is still limited (Eckerd and

Moulton 2011 ). A few recent surveys showed a majority and growing portion of nonprofits

stating that they evaluate their results on beneficiaries and communities mainly in response to