Institutional design and biases in evaluation reports by international organizations

• Published date: 01 May 2024
• Author: Steffen Eckhard,Vytautas Jankauskas,Elena Leuschner
• Date: 01 May 2024
• DOI: http://doi.org/10.1111/puar.13705

RESEARCH ARTICLE

Institutional design and biases in evaluation reports by

international organizations

Steffen Eckhard

1

| Vytautas Jankauskas

1

| Elena Leuschner

2

1

Zeppelin University, Friedrichshafen, Germany

2

Department of Political Science, University of

Gothenburg, Gothenburg, Sweden

Correspondence

Steffen Eckhard, Zeppelin University, Am

Seemooser Horn 20, 88045 Friedrichshafen,

Germany.

Email: steffen.eckhard@zu.de

Funding information

The German Research Foundation, Grant/Award

Number: EC 506/2-1

Abstract

Governments spend hundreds of millions on evaluations to assess the performance

of public organizations. In this article, we scrutinize whether variation in the institu-

tional design of evaluation systems leads to biases in evaluation findings. Biases

may emerge because influence over evaluation processes could enable the bureau-

cracy to present its work in a more positive way. We study evaluation reports pub-

lished by nine international organizations (IOs) of the United Nations system. We

use deep learning to measure the share of positive assessments at the sentence

level per evaluation report as a proxy for the positivity of evaluation results. Analyz-

ing 1082 evaluation reports, we find that reports commissioned by operative units,

as compared to central evaluation units, systematically contain more positive assess-

ments. Theoretically, this link between institutional design choices and evaluation

outcomes may explain why policymakers perceive similar tools for evidence-based

policymaking as functional in some organizations, and politicized in others.

Evidence for practice

•We show that the institutional design of evaluation systems in international

organizations is associated with differences in evaluation findings (i.e., biases).

Across more than 1000 evaluation reports, those commissioned by operative

administrative units are on average systematically more positive than those

commissioned by central evaluation units.

•Whether the senior management of the administration itself, or the organiza-

tion’s governing body (here, member states) controls evaluation system

resources (staff, budget, agenda) does not systematically affect how positive

evaluation findings are.

•Our measurement relies on computational text analysis of evaluation reports,

which enables automatic extraction of performance measures from evaluation

reports, often hundreds of pages long.

•Publications such as this will likely increase the pressure on evaluators, because it

is now clear that text-based performance statements can be quantified, which

enables benchmarking. When designing evaluation systems, decision-makers

should therefore focus on enlarging the institutional distance between evaluators

and those evaluated toensure independent evaluation and objective results.

INTRODUCTION

Over the past decades, evaluation has evolved to become

the main tool to assess the performance and impact of

public sector organizations. Following-up on the features

of modern public management, many national govern-

ments around the globe now commit to evidence-based

policy-making and have installed novel units or entire

agencies for the conduct of evaluation.

1

A common defi-

nition describes evaluation as “an assessment, conducted

Received: 11 September 2022 Revised: 28 June 2023 Accepted: 30 June 2023

DOI: 10.1111/puar.13705

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribu tion and reproduction in any medium, provided the

original work is properly cited.

© 2023 The Authors. Public Administration Review published by Wiley Periodicals LLC on behalf of American Society for Public Administration.

560 Public Admin Rev. 2024;84:560–573.

wileyonlinelibrary.com/journal/puar