Does the 'blindness' of peer review influence manuscript selection efficiency?

• Author: Laband, David N.

The results of the experiment do indicate, contrary to the expectation of some (including myself), that the refereeing process does have an effect on which papers we decide to publish. I believe it was this finding, despite the ambiguous findings regarding the nature of any biases in the decisions, which motivated the vote by our Board of Editors.

Orley Ashenfelter |1, 594~

1. Introduction

   According to Ashenfelter |1~, the recent decision by the Board of Editors of the American Economic Review to adopt a double-blind review policy was based, in his option, on the findings reported by Blank |3~. In her words:

   . . . there are significant differences in acceptance rates and referee ratings between single-blind and double-blind papers. Most strikingly, double-blind papers have a lower acceptance rate and lower referee evaluations. In addition, double-blind reviewing results in different patterns of acceptance rates and referee ratings by institutional rank of author |3, 1042~.

   Although we find Blank's results compelling, it is difficult to draw solid conclusions from them. Perhaps this is what Ashenfelter meant when he referred to the "ambiguous" findings regarding the nature of any biases. The issue of scientific concern vis-a-vis type of review process employed by journal editors is the severity of type-I and type-II errors. Do editors employing a single-blind review process systematically publish more papers that have little or no impact on the profession and/or fail to publish more truly good papers than do editors of double-blind journals?

   Blank's findings do not provide certain relevant information with respect to the type-I/type-II error problem that arguably plagues single-blind reviewing but not double-blind reviewing. One must have information regarding how the marketplace for scientific ideas responds to published papers in order to gauge the severity of both types of errors. Without knowing the fate of manuscripts rejected for publication in Blank's (or any other researcher's) sample, the severity of the type-II error problem cannot be determined. However, drawing from a large sample of papers published in the top economics journals in 1984, we are able to investigate the degree to which journals employing a single-blind review process suffer from the type-I error problem (i.e., publish papers that are revealed not to have the impact that might reasonably have been expected).

   Both types of errors might characterize a single-blind review process for at least two reasons. First, a reviewer with knowledge of the author's identity might economize on his (her) refereeing costs by substituting the already-revealed value of the author's average contribution in previous papers for his evaluation/forecast of the marginal contribution contained in the paper under consideration. Second, personal characteristics of the author (gender, institutional affiliation, friendship with the reviewer, race, intellectual conformity with the reviewer, etc.) may weigh more heavily in a reviewer's evaluation of the publishability of a manuscript than the reviewer's forecast of the marginal contribution contained therein.(1)

   H. Data, Methodology and Findings

   Data

   We compiled detailed information on 1,051 articles (excluding comments, replies, notes and book reviews) published in 28 top economics journals in 1984.(2) Specifically, we identified citations to each article, as listed in the Social Sciences Citation Index for the five years following publication; article-specific characteristics (length in AER-equivalent pages and whether or not it was published as a lead article); characteristics of the authors (age and professional affiliation at the time the paper was published, their Ph.D.-granting institution, gender, and cumulative stock of citations (to all previous work) in the five years prior to 1984, as a proxy for author reputation or quality); characteristics of editors, co-editors and associate editors (institutional affiliations and Ph.D.-granting institutions); and the type of review process employed by each journal. Variable means and standard errors for the entire sample and the samples of double-blind and single-blind reviewed papers are reported in Table I.(3)

   Table I. Means and Standard Errors for All Variables by Review Process Entire Double-Blind Single-Blind Variable Sample(*) Review(*) Review(*) Citations 1985-89 7.057 6.733 7.333 (11.173) (10.520) (11.704) Length 11.721 10.929 12.396 (7.301) (5.552) (8.461) Lead Article 0.091 0.099 0.085 (0.288) (0.299) (0.279) Author(s)Stock of Citations 1979-83 132.382 109.537 151.882 (302.683) (259.188) (334.374) Authors' Mean Age 38.221 37.982 38.425 (7.737) (7.543) (7.900) Review Process, Double-Blind = 1 0.461 1.000 0.000 (0.499) Gender, Woman = 1 0.081 0.072 0.088 (0.273) (0.259) (0.284) Journal Quality Index 49.370 38.543 58.612 (23.926) (14.028) (26.613) N 1051 484 567 * Standard errors in parentheses. We note, without comment at this stage, that papers published in double-blind reviewed journals are shorter and attract fewer citations, on average, than papers published in single-blind reviewed journals. In addition, the average citation stock of authors of papers published in single-blind journals is nearly 50 percent greater than for authors of papers published in journals employing a double-blind review process.

   Methodology

   Part of the difficulty in interpreting findings that different editorial practices result in different outcomes for authors is that there is no well-articulated theory of the editorial process. We suspect that most academic scientists believe that editors should attempt to maximize the expected impact that articles published in their journals have on subsequent scientific thought within the relevant community of scholars. This implies that a scientific manuscript be evaluated solely on the basis of expected marginal contribution to scientific knowledge, not on the basis of non-substantive criteria.

   On the other hand, one could well imagine a theory of the editorial process that is governed by the principle of editorial favoritism towards former and current graduate students, colleagues, faculty at the "elite" schools, etc. Indeed, charges of editorial favoritism have been raised in many a private conversation among economists. Individual scholars may entrepreneur (and, in so doing, become editors of) scientific journals as a means of maximizing their own influence within a personally-relevant community of scholars. In this world, editors selectively supply page-space in their journals to prospective authors in exchange for past, present and/or future considerations that both parties agree upon. Editors include personal well-being, as well as the value of scientific knowledge produced in their decision calculus. Manuscripts are not necessarily, or even probably, evaluated on the basis of the expected marginal scientific contributions contained therein.

   Yet a third possibility, suggested by the reviewer, is that journal editors have certain idiosyncratic biases/preferences regarding what they feel are important areas of scientific...