Do ratings of firms converge? Implications for managers, investors and strategy researchers

• Date: 01 August 2016
• Published date: 01 August 2016
• Author: Aaron K. Chatterji,David I. Levine,Rodolphe Durand,Samuel Touboul
• DOI: http://doi.org/10.1002/smj.2407

Strategic Management Journal

Strat. Mgmt. J.,37: 1597–1614 (2016)

Published online EarlyView 11 August 2015 in WileyOnline Library (wileyonlinelibrary.com) DOI: 10.1002/smj.2407

Received 19 August 2013;Final revisionreceived 9 October 2014

DO RATINGS OF FIRMS CONVERGE? IMPLICATIONS

FOR MANAGERS, INVESTORS AND STRATEGY

RESEARCHERS

AARON K. CHATTERJI,1*RODOLPHE DURAND,2DAVID I. LEVINE,3and

SAMUEL TOUBOUL4

1Duke University, Fuqua School of Business, Durham, North Carolina, U.S.A.

2HEC Paris, Center for Research on Society and Organizations, France

3Haas School of Business, University of California, Berkeley California, U.S.A.

4IPAG Business School, Department of Corporate Strategy & Finance, Paris, France

Research summary: Raters of rms play an important role in assessing domains ranging from

sustainability to corporate governance to best places to work. Managers, investors, and scholars

increasingly rely on these ratings to make strategic decisions, invest trillions of dollars in capital,

and study corporate social responsibility (CSR), guided by the implicit assumption that the

ratings are valid. We document the surprising lack of agreement across social ratings from six

well-established raters. These differences remain even when we adjust for explicit differences in

the denition of CSR held by different raters, implying the ratings have low validity. Our results

suggest that users of social ratings should exercise caution in interpreting their connection to

actual CSR and that raters should conduct regular evaluationsof their ratings.

Managerial summary: Ratings of corporate social responsibility (CSR) guide trillions of dollars

of investment, but managers, investors, and researchers know little about whether these ratings

accurately measure CSR. In practice,there are examples of highly rated rms becoming embroiled

in scandals and the same rm receiving sharply different ratings from different rating agencies.

We evaluate six of the leading raters and nd little overlap in their assessments of CSR. This

lack of consensus suggests that social responsibility is challenging to measure reliably and that

users of these ratings should be cautious in drawing conclusions about rms based on this data.

We encourage the rating agencies to regularly validate their data in an effort to improve the

measurement of CSR. Copyright © 2015 John Wiley & Sons, Ltd.

INTRODUCTION

How much do we really know about corporate

social responsibility (CSR)? Though many man-

agers, investors, and scholars have embraced this

concept, the ratings most often used to assess

CSR have rarely been evaluated. If these ratings

are invalid, then trillions of dollars of capital is

Keywords: corporate social responsibility; ratings; corpo-

rate governance; socially responsible investing; perfor-

mance measurement

*Correspondence to: Aaron K. Chatterji, Fuqua School of Busi-

ness, Duke University, 1 Towerview Road, Durham, NC 27708,

U.S.A. E-mail: ronnie@duke.edu

Copyright © 2015 John Wiley & Sons, Ltd.

potentially being misallocated and numerous aca-

demic ndings may also not be valid.

In this study, we assess the convergent validity

(that is, agreement) of six well-established social

ratings. We nd that these raters exhibit low con-

vergence in their assessments of CSR.1This lack

of agreement is not just due to announced differ-

ences in raters’ theorization of CSR; for example,

if they measure performance relative to an industry

1When discussing the behavior of raters, we use the term conver-

gence. When referring to the rating they provide, we use the term

convergent validity. We do not wish to imply that convergence

implies a particular time trend. We apply this term to describe

overlap across ratings systems at a particular point in time.

1598 A. Chatterji et al.

group or in absolute terms. Instead, the low agree-

ment implies all or almost all of the ratings have low

validity. This result has important implications for

managers, investors, and researchers who use these

ratings.

Many managers spend signicant time and

resources on CSR activities. For example, analysts

claim that nearly every Fortune 500 company

releases some kind of sustainability report.2Eight

thousand rms have signed the UN Global Com-

pact as a sign of their commitment to CSR.3As

CEOs and other top managers respond to growing

pressure from multiple stakeholders over social

issues (Bansal and Roth, 2000; Crilly, Zollo,

and Hansen, 2012), high-prole and publically

disseminated social responsibility ratings take on

even greater importance. But if the ratings are

not actually valid and cannot consistently identify

socially responsible rms, then the hypothesized

benets of CSR cannot occur. For example, if

managers cannot deduce whether their low rating is

due to poor operations and performance, a different

conceptualization of CSR than the raters, or simply

poor measurement (Gray, 2010; Margolis and

Walsh, 2003), then they will be unable to craft the

appropriate response. In the worst-case scenario, if

rms expend resources to achieve high scores on

invalid metrics, then even well-intended attention

to social metrics reduces social welfare.

Similarly, investors face serious challenges if

metrics are invalid. If the enormous amount of

socially responsible investment (SRI), approx-

imately one out of every nine dollars in the

United States4and one out of every six dollars in

Europe (Cortez, Silva, and Areal, 2012), is being

erroneously allocated to rms, then it implies sig-

nicant inefciencies in global capital markets. If

the organizations that rate the social performance of

enterprises, referred to as “raters” or “SRI raters” in

our study, cannot discern which rms are socially

responsible (Delmas, Etzion, and Nairn-Birch,

2013; Entine, 2003; Hawken, 2004), then SRI will

not direct capital toward the most responsible rms.

Thus, low convergent validity ensures the promise

of “doing good and doing well” will be unfullled.

2Kanani (2012).

3“From Fringe to Mainstream: Companies Integrate CSR Initia-

tives into Everyday Business.” (http://knowledge.wharton.upenn.

edu/article/from-fringe-to-mainstream-companies-integrate-csr-

initiatives-into-everyday-business/)Last accessed July 21, 2013.

4US SIF Foundation, 2012 Report on Sustainable and Responsi-

ble Investing Trends in the United States.

Academics should also be concerned about the

convergent validity of SRI ratings. The academy

has produced scores of articles on CSR and SRI

over the past two decades (Orlitzky, Schmidt, and

Rynes, 2003), with growing interest in recent years.

For example, from 1994 to 2008, seven articles

published in SMJ relied on data from just one of

our SRI raters (KLD). From 2009 to 2013, 19 arti-

cles used KLD data and six articles employed other

ratings we examine (FTSE4Good, Innovest, DJSI

or Asset4). Notably, inuential research has exam-

ined the effects of CSR on returns for investors

and the cost of capital for managers (Galema,

Plantinga, and Scholtens, 2008; Waddock, 2003).

Other research has explored the drivers of CSR,

such as prot-maximizing responses to heteroge-

neous consumer preferences (Mackey, Mackey, and

Barney, 2007), imitation among rms, or a depar-

ture from prot-maximizing behavior to satisfy

managers’ private goals (Devinney, 2009; Marquis,

Glynn, and Davis, 2007).

However, despite this growing interest in CSR,

little research examines whether raters measure

CSR accurately (Delmas et al., 2013; Sharfman,

1996). If these metrics are invalid or are inconsis-

tently applied across raters, then scholars who con-

duct analysis using one rating scheme risk drawing

conclusions that are not accurate. Moreover, if there

is systematic measurement error in SRI ratings, then

scholars may report effects, for example, the posi-

tive or negative effect of CSR on rm performance,

that are not true.

In short, it is crucial for managers, investors

and academics to know the validity of social

ratings and understand the dynamics driving

convergence across raters. In this article, we rst

document that the ratings of six major social

raters— KLD, Asset4, Calvert, FTSE4Good, DJSI,

and Innovest— have fairly low correlations with

each other. We then show that the correlation does

not systematically increase when we adjust for

announced differences in raters’ theorization of

CSR. Our results imply that SRI raters not only

do not agree on one denition of responsibility

(their “theorizations” of CSR differ), but also that

raters may measure the same construct in different

ways (the “commensurability” of CSR is low). Our

ndings suggest that consumers of this data should

interpret SRI ratings with caution and validate

these ratings before drawing strong conclusions

about CSR.

Copyright © 2015 John Wiley & Sons, Ltd. Strat. Mgmt. J.,37: 1597–1614 (2016)

DOI: 10.1002/smj