Does paying referees expedite reviews?: Results of a natural experiment.

• Author: Thompson, Gary D.

1. Introduction

   Scientific journals have employed peer review to appraise the quality of manuscripts for over three centuries (Zuckerman and Merton 1971). Referees providing peer reviews generally do so voluntarily as part of their professional duties; they often receive no payment for reviewing manuscripts. Referees' rewards for timely, high-quality reviews are usually limited to nonpecuniary benefits, such as standing in good stead with editors and recognition as a referee for a prestigious journal.

   When requesting peer reviews, journal editors place their trust in referees to provide high-quality, timely reviews. While editors can try to choose referees who will provide high-quality reviews, editors are able to provide few incentives to elicit careful, timely reviews or impose penalties on delinquent referees (Freedman 2000). Referees write and submit reports in virtual anonymity whether reviews are single or double blind. If referees shirk or lag in completing their reviews, they usually suffer little damage to their public reputation.

   Some economics journals offer payments as an incentive to referees for providing on-time reviews. Somewhat surprisingly, empirical research examining the efficacy of payments to encourage punctuality is quite limited. Apparently, only one empirical study has been conducted. Hamermesh (1991, 1994) provides cross-sectional evidence from a sample of 50 manuscripts submitted to seven journals in November 1989. Journals offering payments enjoyed a larger proportion of referee reports completed on time.

   It is perhaps even more surprising that there is so little empirical research on the efficacy of referee payments given current review and publication lags at some economics journals. The increased time between the submissions of manuscripts to publication in economics journals over the last four decades has been documented thoroughly (Marshall 1959; Coe and Weinstock 1967; Yohe 1980; Laband, Maloney, and McCormick 1990; Trivedi 1993; Chong 2001; Ellison 2002). Ellison attributes about one quarter of the increase in the submission-to-publication lag to increased times between submission and the first editorial response, the first-response time (FRT). Azar (2004) argues that delays in FRT actually account for more of the total lag if FRTs of manuscripts previously rejected by journals other than the publishing journal are taken into account. Most journals offering referee payments do so only in the first round of reviews in an effort to reduce FRT.

   The possible benefits of the slowdown in FRT to authors, referees, editors, and journal readers have been examined extensively by Azar (2005a, b, 2007, 2008). Most authors prefer quicker FRTs, and reductions in FRTs should lead to quicker dissemination of scientific results. However, shorter FRTs may encourage submission of lower-quality manuscripts to high-quality journals, resulting in costs to referees and editors. With more manuscripts to handle, rejection rates may climb, and authors may have to submit their manuscripts to more journals before manuscripts are ultimately accepted for publication. This article makes no judgment about the social welfare of reducing FRTs. (1) Rather, it tackles the more modest question of whether payments induce referees to expedite their reviews and thereby reduce FRTs.

   Previous theoretical models for paying referees distinguish the participation rate--the rate at which referees agree to review manuscripts--from the review time. Chang and Lai (2001) find possible multiple equilibria in which payments may be offered by some journals because not all journals are prestigious enough to attract referees without compensation. Engers and Gans (1998) endogenize both the participation rates and the review time in considering the equilibrium level of payments. The equilibrium payment level is zero because neither the referees nor the editors fully internalize the effects of their decisions on journal quality. Hamermesh's (1991) appointment-book model also treats the participation rates and review times separately. Although the model does not unambiguously generate hypotheses regarding refusal rates and review times, the role of payments to referees is to elicit "queue-jumping," that is, to influence the order in which voluntary tasks are completed by referees. These theoretical models highlight a salient feature of the market for referee services: Because referees provide their services voluntarily, there is no pricing mechanism to allocate effort.

   Many economists would expect the answer to the question of whether payments to referees for on-time reviews can expedite first-round reviews to be self-evident: Payment for what was previously a voluntary activity should not perversely affect effort and performance (Arrow 1972). (2) The only question might plausibly be the degree to which payments can expedite reviews. However, some empirical evidence suggests that when voluntary economic activities--giving blood, volunteering to work for public or private institutions, and collecting donations for charity, for example--are rewarded with relatively low payment levels, low-paid performance is inferior to voluntary performance. In their analysis of a cross-section of Israeli high school students collecting donations for charity, Gneezy and Rustichini (2000) find statistically significantly smaller donations collected by students to whom relatively low payments were offered; students volunteering and students receiving higher payments collected higher levels of donations. Econometric evidence from a cross section of workers in the Swiss Labor Force Survey indicates that relatively low levels of financial compensation for voluntary activities result in fewer hours volunteered per month compared to hours volunteered by workers who received either no compensation or higher levels of compensation (Frey and Goette 1999). These two studies suggest that paying for what was once a voluntary activity like refereeing manuscripts could produce unexpected results, especially if the payment is low. Different types of payments may elicit different effects on previously unpaid or voluntary tasks. In the two studies mentioned, levels of payment varied discretely (Gneezy and Rustichini 2000) or continuously (Frey and Goette 1999). In the natural experiment considered here, a single payment level was established. Whether a single payment level can expedite first-round reviews remains an open question.

   The purpose of this study is to provide some empirical evidence about the effects of payments to referees for a single field journal, the Journal of Agricultural and Resource Economics (JARE), through time. The question addressed is as follows: Do payments significantly reduce first-response times (FRT)? We have information on these response times both before and after payments were implemented at JARE. Hence, we can complement Hamermesh's (1994) cross-sectional comparisons of journals with and without payments by making time-series comparisons of the efficacy of payments to referees of the same journal. No new theory to explain on-time behavior is offered. Instead, new data from a natural experiment and econometric analysis assessing the impacts of payments on review and FRTs are presented.

2. The Natural Experiment

   The natural experiment generating the data consists of editorial records for JARE before payment to referees and after implementing payments. JARE is the journal of the Western Association of Agricultural Economics (WAEA). The editorial duties of JARE typically rotate every three years. An editorial team of an editor and three co-editors assumed responsibility for JARE from April 2000 to April 2003. The editorial period of our data begins April 2000. For purposes of continuity, all initial submissions received through March 2003 were handled by the same editors until the first editorial response. All subsequent editorial correspondence and revised manuscripts were forwarded to the new editors, whose duties began in April 2003.

   In July 2001, the executive council of WAEA, the governing body responsible for approving new uses of the WAEA budget, voted to approve a payment of $50 for on-time first-round reviews of manuscripts submitted to JARE. On-time reviews were taken to be those completed within six weeks of the referee agreeing to the editor's request to review a manuscript. This natural experiment generated 15 months' worth of reviews with no payment-April 2000 through June 2001--and about 24 months' worth of reviews under the payment regime. All reviews were double blind.

   The protocol for soliciting reviews, whether with or without payments, was to send prospective referees an electronic mail message soliciting a review. After the referee agreed to review the manuscript, a copy of the manuscript with cover letter was sent either electronically or by regular mail to the referee. Although JARE policy encouraged electronic submission, many authors submitted manuscripts by mail. Manuscripts not submitted electronically were usually sent by courier to referees residing outside of North America to reduce lag times in foreign mail delivery. Nearly all referees' reports were submitted by electronic mail to the editors, facilitating exact dating of reports. In the majority of cases, the determination of an on-time submission of the referee's report was clear-cut because reports were submitted before the...