Performance feedback in competitive product development

• Author: Daniel P. Gross
• Date: 01 May 2017
• DOI: http://doi.org/10.1111/1756-2171.12182
• Published date: 01 May 2017

RAND Journal of Economics

Vol.48, No. 2, Summer 2017

pp. 438–466

Performance feedback in competitive

product development

Daniel P. Gross∗

Performance feedback is ubiquitous in competitive settings where new products are developed.

This article introducesa fundamental tension between incentives and improvement in the provision

of feedback. Using a sample of 4294 commercial logo design tournaments, I show that feedback

reduces participation but improves the quality of subsequent submissions, with an ambiguous

effect on high-quality output. Toevaluate this trade-off, I develop a procedure to estimate agents’

effort costs and simulate counterfactuals under alternative feedback policies. The results suggest

that feedback on net increases the number of high-quality ideas produced and is thus desirable

for a principal seeking innovation.

Performance feedback is pervasive in industries where new products are developed: workers

pitch ideas to managers and clients, who provide critiques and direction; prototypes are tested at

Research and Development (R&D) labs, with focus groups, and in public demonstrations; and

consumers are the ultimate arbiters of value. Despite widespread use, the effects of feedback

on innovation in competitive environments are not fully understood: feedback is argued to be

essential to improving innovation in single-agent contracts (Manso, 2011), but research on the

effects of performance disclosure in routine competitive settings (e.g., Ederer, 2010) has shown

that it can weaken incentives by revealing asymmetries between competitors.

This article studies the tension between incentives and improvement in feedback provision,

which is of fundamental importance to firms in creative or research industries and other orga-

nizations procuring or funding innovation. This tension is intrinsic to product development and

is especially stark in formal competitions such as innovation contests (e.g., Taylor, 1995; Che

and Gale, 2003; Terwiesch and Xu, 2008), which are undergoing a renaissance in the private,

∗Harvard Business School and NBER; dgross@hbs.edu.

I thank Ben Handel, Przemek Jeziorski, Gustavo Manso, John Morgan, Denis Nekipelov, and the Editor and referees

for comments that improved this article, as well as to the Harvard Business School Division of Research and Faculty

Development and NSF Graduate Research Fellowship grant no. DGE-1106400 for financial support. Limited short

segments of the text may be similar to passages from another of my articles (“Creativity under Fire: The Effects of

Competition on Creative Production”), which uses a different data set from the same setting. A previous version of this

article circulated under the title “Interim Feedback in Tournaments for Ideas: Trading OffParticipation for Quality.” A

online Appendix is available at http://www.dpgross.com/. All errors are my own.

438 C2017, The RAND Corporation.

GROSS / 439

public, and nonprofit domains (Williams, 2012).1It also surfaces in more traditional R&D con-

tracting environments: organizations seeking to procure new technology often issue requests for

proposals and take bids over multiple rounds, with opportunities for feedback in between.In these

settings, the sponsoring organization has better information on performance than participants or

can compel interim disclosure of progress while a competition is underway.2Should it make this

information known?

A similar tension is present in noninnovation organizational settings, where performance

appraisals serve the dual purposes of employee development and evaluation for tournament-like

promotion and retention (Beer, 1987). The observation that performance evaluation is a near-

universal practice in organizations has motivated a nascent economics literature on the effects

of feedback in promotion tournaments (Gershkov and Perry, 2009; Aoyagi, 2010; Ederer, 2010;

Goltsman and Mukherjee, 2011), but this literature is short on empirical evidence and, with the

exception of Wirtz (2014), does not account for the effects of feedback on agents’ productivity.

Another subliterature (Gill, 2008; Rieck, 2010; Choi, 1991; Bimpikis, Ehsani, and Mostagir,

2014; Halac, Kartik, and Liu, 2015; Mihm and Schlapp, 2016) studies disclosure policies in the

context of patent races and innovation contests, but it, too, is exclusively theoretical and has

excluded the possibility of feedback-driven improvement.

In this article, I use a large sample of winner-take-all logo design competitions to study

the dual effects of feedback on the quantity and quality of agents’ output. I first show that

feedback causes players to advantageously select into continued participation and improves the

quality of subsequent entries, but disclosure of intense competition discourages effort from even

the top performers. A principal seeking a high-quality product thus faces a trade-off between

participation and improvement in deciding whether to provide its agents with feedback. To better

understand this trade-off, I estimate a structural model of the setting and use the results to

simulate tournaments with alternative feedback mechanisms. The results suggest that feedback

on net increases the number of high-quality designs generated, implying that it can be a valuable

tool for generating innovation despite its adverse effects on incentives.

The article begins by developing a simple, illustrativemodel of a winner-take-all innovation

contest to clarify the forces at play. In this model, a principal seeks a new product design and

solicits candidates through a tournament, awarding a prize to the best entry. Players take turns

submitting ideas, each of which receives immediate, public feedback revealingits quality. Partial-

equilibrium predictions echo previous theoretical findings for other settings: revelation of agents’

performance can be motivating for high performers, but in general it will tend to disincentivize

effort by exposing leaders and laggards. However, feedback also helps agents improve at later

submissions, increasing their productivity at each draw and potentially offsetting the detriment

to participation. Within this framework, I characterize feedback as havingtwo effects: a selection

effect, which drives players with poor reviews or facing fierce competition to quit, and a direction

effect, which guides continuing players toward increasingly better designs.

The article then transitions to an empirical study of 4294 commercial logo design tourna-

ments from a widely used online platform. In these contests, a firm solicits custom designs from

freelance designers, who compete for a winner-take-all prize awarded to the preferred entry.The

contests in this sample typically offer prizes of a few hundred dollars and on average attract

around 35 players and 115 designs. An essential feature of the setting is that the sponsor can

1Prizes have been used for centuries to encourage third parties to solve technological, commercial, and social

problems. In 2010, US federal agencies received broad authority to conduct public prize competitions, with rewards

ranging from status only (nonpecuniary) to tens of millions of dollars (OSTP, 2014).

2In practice, they often do. In the 2006 Netflix contest to develop an algorithm that predicts users’ movie ratings,

entries were immediately tested and scored,with the results posted to a public leaderboard. In the US Defense Advanced

Research Projects Agency’s 2005 and 2007 prize competitions to develop autonomous vehicles, participants had to

publicly compete in timed qualifying races before moving on to a final round. Although these examples involve large

stakes, interim scoring is also common in lower-stakes settings, such as architecture competitions or coding contests

(Boudreau, Lacetera, and Lakhani, 2011; Boudreau, Lakhani, and Menietti, 2014).

C

The RAND Corporation 2017.