P‐Hacking, P‐Curves, and the PSM–Performance Relationship: Is There Evidential Value?

• Published date: 01 March 2021
• Author: Dominik Vogel,Fabian Homberg
• Date: 01 March 2021
• DOI: http://doi.org/10.1111/puar.13273

Research Article

Abstract: Recent developments in the social sciences have demonstrated that we cannot uncritically aggregate the

published research on a particular effect to conclude about its presence or absence. Instead, questionable research

practices such as p-hacking (conducting additional analyses or collecting new data to obtain significant results) and

selective publication of significant results can produce a body of published research that misleads readers even if it

contains many significant results. It is, therefore, necessary to assess the evidential value of the research on a certain

effect; that is, one must rule out that it is the result of questionable research practices. We introduce the p-curve method

to public administration research and apply it to the research on the relationship between public service motivation

(PSM) and individual performance, to demonstrate how the evidential value of a body of published research can be

assessed. We find that this particular literature contains evidential value.

Evidence for Practice

• Public servants with high public service motivation show better performance in the workplace.

• However, a set of significant findings on a particular effect is not necessarily an indicator that this effect

actually exists, and as evidence-based decision-making becomes more popular in public organizations,

decision-makers need to rely on trustworthy and unbiased evidence.

• Introducing and applying the p-curve method, our findings reveal that the literature on the effect of

public service motivation on individual performance is unbiased, that is, that it seems not to be based on

p-hacking and publication bias and, thus, provides a good foundation for evidence-based decisions in public

organizations.

Notwithstanding the efforts to enhance

transparency in leading journals, with

the aim of safeguarding the “quality and

integrity of the science we use to build knowledge

in public administration” (Perry2017), the recent

past has seen rising levels of distrust in the results

published by scholars (Hubbard2015). This

phenomenon is driven not only by the general

public’s increased distrust of scientific evidence on

a broad range of topics, from climate change to

dietary recommendations, but also the heightened

skepticism among scientists themselves in multiple

disciplines. The most prominent example of such

an intra-disciplinary challenge is the “crisis of

confidence” or “replication crisis” in psychology

(Pashler and Wagenmakers2012). In light of the

recent developments in public administration

and the stronger links forged with psychology

(Grimmelikhuijsen et al.2017; Jilke, Meier, and

van Ryzin2018), the replication crisis has become

increasingly relevant for public administration

research.

Besides cases of actual fraud, the starting point

of the replication crisis in psychology was that

only 39 out of 100 studies could be replicated in

a large-scale replication attempt (Open Science

Collaboration2015). This study evoked the long-

standing criticism (e.g., Ioannidis2005) of the use

of questionable research practices with additional

evidence. It seemed to establish that such practices

result in highly inflated type I error rates (wrongly

concluding that there is an effect when there is

actually none). Such practices include hypothesizing

after the results are known (HARKing), conducting

additional analyses or collecting new data to obtain

significant results (p-hacking), and selectively

reporting significant results. These behaviors

naturally lead to the publication of literature that

overwhelmingly consists of significant results

(publication bias) (Earp and Trafimow2015). As

Starbuck(2016, 172) has stated, “[n]ot only are

HARKing and p-Hacking widespread, but sad to

say, editors, reviewers, and colleagues often advise

researchers to use these practices.”

P-Hacking, P-Curves, and the PSM–Performance

Relationship: Is There Evidential Value?

Dominik Vogel Fabian Homberg

University of Hamburg Department of Business & Management, LUISS Guido

Carli University

Fabian Homberg is Associate Professor

of Human Resource Management and

Organizational Behavior at LUISS Guido

Carli University, Rome, Italy. His current

research interests include public service

motivation and incentives in private- and

public-sector organizations.

Email: fhomberg@luiss.it

Dominik Vogel is Assistant Professor

of Public Management at the University

of Hamburg, Germany. His research

focuses on the motivation of public

employees, leadership, and human resource

management in the public sector, interaction

of citizens with public administrations, and

performance management.

Email: dominik.vogel-2@uni-hamburg.de

Public Administration Review,

Vol. 81, Iss. 2, pp. 191–204. © 2020

TheAuthors.

Public Administration Review

published by Wiley Periodicals LLC on

behalf of The American Society for Public

Administration.

DOI: 10.1111/puar.13273.

191

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium,

provided the original work is properly cited.