Do Public Employees “Game” Performance Budgeting Systems? Evidence From the Program Assessment Rating Tool in Korea

• Published date: 01 July 2018
• Author: Hosung Sohn,Kwang Bin Bae
• Date: 01 July 2018
• DOI: 10.1177/0275074016689322
• Subject Matter: Articles

https://doi.org/

American Review of Public Administration

2018, Vol. 48(5) 458 –475

© The Author(s) 2017

Reprints and permissions:

sagepub.com/journalsPermissions.nav

DOI: 10.1177/0275074016689322

journals.sagepub.com/home/arp

Article

Introduction

Since the influential work of Osborne and Gaebler (1992),

many government reforms have been adopted to hold gov-

ernment accountable for its performance. The Hoover

Commission, Planning Programming Budgeting System,

Management by Objectives, and Zero-Based Budgeting are

the examples of government reforms that aim to promote

government performance. One of the major reforms is the

Government Performance and Results Act (GPRA) of 1993.

The purpose of the Act is to promote effectiveness and effi-

ciency of federal programs and spending by measuring pro-

gram performance, providing publicly the reports on their

progress, and holding agencies accountable for achieving

program goals. The GPRA, however, failed to produce infor-

mation relevant to improving program performance (Dull,

2006; Gilmour & Lewis, 2006b). The Program Assessment

Rating Tool (PART), instituted by President George W. Bush

and ran through the Office of Management and Budget, is

one of the policies introduced for the purpose of overcoming

the limitation of the GPRA. As with the GPRA, one of the

main goals of PART is to rigorously and systematically eval-

uate the effectiveness of public programs and establish a sys-

tematic link between information provided by program

assessments and budget processes.

The adoption of PART is based on the argument that incor-

porating performance-based system in the budgeting process

and linking the performance to budgeting would promote the

performance of public programs (Broom & McGuire, 1995;

Grizzle & Pettijohn, 2002; Joyce, 1993). For this reason, previ-

ous studies focus on whether the PART information, indeed, is

used for linking performance information to budgeting

activities (e.g., Gilmour & Lewis, 2006a; Lavertu & Moynihan,

2012). Note that the usefulness of the PART information hinges

critically upon whether the information provided by agencies

truly reflects program performance. PART may not provide

adequate information that is crucial for evaluating performance

of programs, however, because of the special features inherent

in the public sector. Such features include, but are not limited to,

multiple principals and multiple goals (Dixit, 2002), ambiguity

in agency goals, and difficulty in quantifying the measures of

performance (Frederickson & Frederickson, 2006; Radin,

2000). Under these features, extracting veritable performance

measures from the public sector is extremely difficult, and as

such, the PART information may not appropriately reflect the

performance level of public programs.

Also, performance management systems such as PART

may give rise to one particular form of unintended conse-

quences: gaming. Program administrators, who are likely to

be budget maximizing agents, may behave strategically

under the PART system, in particular, because performance

is explicitly linked to the budgeting process (Niskanen,

1975). There are many reasons that a system like PART may

induce program managers to engage in gaming the system.

First, the primary measure of success in the public sector

may be determined by whether an agent has secured its bud-

get or not. As a consequence, the performance management

1Korea Institute for Health and Social Affairs, Sejong, South Korea

2North Carolina Central University, Durham, NC, USA

Corresponding Author:

Kwang Bin Bae, Department of Public Administration, North Carolina

Central University, Durham, NC 27707, USA.

Email: kbae@nccu.edu

Do Public Employees “Game” Performance

Budgeting Systems? Evidence From the Program

Assessment Rating Tool in Korea

Hosung Sohn1 and Kwang Bin Bae2

Abstract

We examine whether performance budgeting systems such as the Program Assessment Rating Tool (PART) induce public

employees to engage in “gaming” behavior. We propose an algorithm for detecting gaming behavior that makes use of the

discrete nature of the PART system in Korea (KPART) and the revealed patterns of the distribution of the KPART scores.

By employing the test developed by McCrary, we find suspicious patterns in the density of the KPART scores and evidence

points to the fact that manipulation is prevalent in the KPART system. Our analysis suggests that public employees are

sensitive to negative incentives and that great care must be taken when designing performance budgeting systems.

Keywords

performance-based budgeting system, strategic behavior

Sohn and Bae 459

system under PART can be considered as a high-stakes game

because PART enables decision makers to attach budgetary

consequences. Second, PART is distinct from the GPRA in

the sense that PART renders a judgment on whether pro-

grams are effective (Breul, 2007). And because program

managers are sensitive to the judgment they receive, they

have an incentive to game the system in an effort to receive

positive judgment. Third, it is the program managers who

bear the burden of proof on whether the programs are effec-

tive (Heinrich, 2012). This, in turn, implies that the program

managers are aware of the evaluation factors that contribute

to a positive rating. The system, therefore, provides more

room for program managers to engage in undesirable behav-

iors. This is especially likely as principals cannot effectively

determine whether the information provided by the agents

are true or not because of information asymmetry (i.e., the

principal–agent problem). Finally, in the context of the PART

system, the raw score required for a program to receive a

positive rating is oftentimes preannounced. If this is the case,

the system is likely to create incentives for program manag-

ers to engage in manipulating their performance because the

managers are aware of the eligibility cutoff.

If, in fact, gaming is prevalent in the PART system, it is

indisputable that the information provided in PART will nei-

ther be sufficient for correctly evaluating the performance of

government programs, nor appropriate for use in rewarding or

punishing program administrators. In this article, therefore, we

examine whether gaming behavior is prevalent in the PART

system. To fulfill our goal, we use the institutional setting in

South Korea. The general structure of the PART system in

Korea (KPART) is simple—if the score a program receives is

below a certain threshold, the program gets a penalty in the

form of budget cuts. Studying the KPART system is favorable

for investigating the gaming behavior because the system is

likely to induce strategic behaviors given that many of the fea-

tures mentioned above are inherent in the KPART system.

Note that this study uses the “historical” setting to examine the

gaming behavior of public officials, as KPART (not to men-

tion PART) is no longer implemented in Korea. The purpose

of our study, however, is to identify the prevalence of such

behavior—a difficult task—using novel statistical methods,

rather than examining the limitation of the KPART system. We

therefore believe that studying KPART—although no longer

implemented—will provide many lessons for policymakers in

the United States as well as around the world in improving

performance budgeting systems.

Detecting the prevalence of manipulation, however, is eas-

ier said than done for many reasons. It is obvious that if gam-

ing is committed during the evaluation process, it is likely that

actions among program administrators are conducted covertly.

Such behavior might be disclosed if a whistle blower divulges

the cheating. Even if this is the case, however, uncovering the

“hard evidence” is still challenging.

In this article, we propose an algorithm for detecting gam-

ing behavior that makes use of the discrete nature of the insti-

tutional setting in Korea and the revealed patterns of the

distribution of the PART scores combined with the nonpara-

metric statistical test developed by McCrary (2008).

Although the method was initially developed for testing the

manipulation of the assignment variable density function in

the context of regression discontinuity design, the method is

very useful in the context of this article because a density

function of the revealed pattern itself is the object of interest.

Using administrative records of the PART scores maintained

by the Ministry of Strategy and Finance and employing the

test developed by McCrary, we find suspicious patterns in

the density of the PART scores. To evaluate the significance

of our results, we also conduct a series of sensitivity tests as

well as falsification tests, and evidence still points to the fact

that gaming is quite likely. Although the strategy used in this

study and the corresponding results do not “prove” that there

are illicit acts, the analysis strongly suggests that perfor-

mance budgeting systems such as PART may induce unin-

tended behavioral distortions.

The rest of the article is organized as follows. We first

provide a review of literature relevant for the purpose of this

study. Next, we present institutional details of the KPART

system. Third, we introduce the data and statistical method

used in this study, followed by a discussion of statistical

results. Finally, we highlight implications derived from our

study and conclude.

Literature Review

Promoting the performance of public programs has been a

major concern among public administrative reformers, and

performance-based budgeting (PBB) system has been sug-

gested as a way to achieve such goal (Joyce, 1997).

Expecting that PBB system would enhance the perfor-

mance of public programs, one of the major performance

management reform initiatives introduced during the Bush

administration is PART. At the initial stage of PART, many

advocated the PART system and anticipated that it would

bring performance information into the budgeting process,

and consequently, leading to a “results-oriented” govern-

ment (e.g., Breul, 2007; Dull, 2006).

Because of its intended purpose (i.e., linking performance

information to budgeting activities and improving program per-

formance), some of the previous studies on PART focus on

whether the PART scores are used properly for affecting budget

choices (e.g., Gilmour & Lewis, 2006a, 2006b; Greitens &

Joaquin, 2010; Lavertu & Moynihan, 2012). While the conclu-

sions as to the question on whether PART promoted the pro-

gram performance are not unanimous among these studies,

every study concludes that the PART information is not exten-

sively used in the budgeting process (Moynihan, 2013).

As a matter of course, research on the effectiveness of

PART is predicated on the belief that the PART score rightly

reflects program performance and that evidence presented by

program managers is of high quality.1 Assessing the quality

of PART information, therefore, should precede the evalua-

tion of PART usage (Joyce, 1997). Deriving quantifiable and