Meta‐Analytic Research in Management: Contemporary Approaches, Unresolved Controversies, and Rising Standards

• Author: Andreas Rauch,James G. Combs,T. Russell Crook
• Date: 01 January 2019
• Published date: 01 January 2019
• DOI: http://doi.org/10.1111/joms.12427

© 2018 John Wiley & Sons Lt d and Society for the Adva ncement of Management Stud ies

Meta-Analytic Research in Management:

Contemporary Approaches, Unresolved

Controversies, and Rising Standards

James G. Combsa,b, T. Russell Crookc and Andreas Rauchd

aUniversity of Ce ntral Florida; bUniversity of O ttawa; cUnive rsity of Tennessee; dThe Uni versity of

Sydney

ABST RACT Early meta-ana lyses in management research sought prima rily to resolve seemingly

confl icting finding s by estimating a relationship’s population-level ef fect size. Since then,

management researchers h ave adopted increasi ngly sophisticated approaches that permit new

theorizing, t esting and comparing sophistic ated models, and identifying bounda ry conditions.

We summarize t hree of these approaches – i.e., qualitative meta-an alysis (QMA), meta-analyt ic

structura l equation modeling (MA SEM), and meta-analyt ic regression analysis (M ARA)

– along with the spe cial issue papers that adopt each approach. We conclude by raisi ng three

unresolved controversies t hat we believe deserve more attention and by offering ou r thoughts

about how to maximi ze a meta-anal ytic study’s chances for publication and impact.

Keywo rds: meta-a nalysis, meta-a nalytic struct ural equation model ling, meta-regres sion,

qualitat ive meta-analysi s

INTRODUCTION

Meta-analysis ha s become essential in the evolution of knowledge about management.

In their review of meta-ana lytic methodological choices, Ag uinis et al. (2011) reviewed

almost 200 meta-ana lyses and noted that they are being published at an increasing rate.

Meta-analyses are a lso read and cited. The average meta-ana lysis in the Ag uinis et al.

(2011) review was cited 8.47 times per year, which is almost twice the number of annual

citations received by the 500 top cited papers in ma nagement (see Patience et al., 2017).

The reason for the popularity of meta-analysis is simple. It allows researchers to aggre-

gate evidence across studies that investigate similar theoretical predictions (e.g., possessing

strategic resources enables firms to enjoy stronger performance – Crook et al., 2008) or

sets of relationships around the same phenomenon (e.g., human resource management

Journal of Man agement Studi es 56:1 Januar y 2019

doi:10. 1111/j om s. 124 27

Address for re prints: James G. C ombs, Della Phil lips Martha Sc henck Chair of America n Private Enterpr ise,

Department of M anagement, Univers ity of Centra l Florida, Orl ando, FL 32816, USA; Telfer School of

Management, Uni versity of Ottawa, Ott awa, Ontario KIN 6N5, C anada (james.combs @ucf.edu).

2 J. G. Combs et al.

© 2018 John Wiley & Sons Lt d and Society for the Adva ncement of Management Stud ies

practices – Jiang et al., 2012). In essence, meta-analysis allows researchers to create

‘super-samples’ or ‘a sample from many samples’ to estimate the direction, size, and

variance of theoretical relationships – estimates that researchers can have confidence

in because they summarize information across many different studies. Meta-analysis is

also increasingly popular because today’s meta-analytic tools allow researchers to go

beyond merely aggregating seemingly conflicting findings around single relationships.

Researchers can inductively build new theory (e.g., Rauch et al., 2014), investigate

whether bodies of research conform to complex systems of simultaneous predictions

(Bergh et al., 2016; Jak, 2015), assess the relative importance of competing predictions

(e.g., Karam et al., 2019), and test theoretical ideas that would be nearly impossible to

investigate in a single study (e.g., Carney et al., 2011).

To support researchers’ ever-increasing need to aggregate and synthesize what is known

about important management phenomena, and to take advantage of meta-analytic tools

to advance and test new theory, the Editors at the Journal of Management Studies envisioned

and supported this special meta-analysis issue. Our purpose in this special issue introduc-

tion is threefold. First, we offer a description of the three major advanced approaches

to meta-analysis as both a summary for those just learning and as a guide for those in

the early design phase of a meta-analysis. We leave it to the many excellent ‘how to’

guides to direct researchers through the mechanics of the three approaches (e.g., Bergh

et al., 2016; Cooper, 2010; Geyskens et al., 2009; Gonzalez-Mulé and Aguinis, 2018;

Hoon, 2013; Jak, 2015; Rauch et al., 2014; Schmidt and Hunter, 2015) and instead focus

on explaining their central purposes, strengths, and challenges. We start with qualitative

meta-analysis (QMA), which synthesizes rich case evidence to both test and build new

theory. We then discuss two quantitative approaches that have become standard practice

in management research – i.e., meta-analytic structural equation modelling (MASEM)

and meta-analytic regression analysis (MARA).

Our second purpose is to summarize the papers in the special issue and highlight how

each contributes to its respective literature. All of the papers in the special issue draw on

one of the three advanced meta-analytic approaches. Thus, we organize our summary of

the special issue papers by meta-analytic approach and use our summaries as examples

to highlight each approaches’ strengths and challenges.

Our final purpose is to draw attention to issues we encountered as editors that we believe

warrant attention from students and practitioners of meta-analysis. For students, there

appears to be unresolved controversy around the use of (a) Fisher’s z transformation, (b)

partial correlation coefficients (i.e., standardized betas), and (c) meta-regression. For me-

ta-analytic practitioners, our sense is that many meta-analyses are rejected due to authors’

failure to adhere to rising standards. We highlight these issues in the Discussion section.

THE ROLE OF META-A NALYSIS IN MANAGEMENT RESE ARCH

The famous Hawthorne studies ( Roethlisberger and Dick son, 1939) sparked the Human

Relations movement and generated theory anticipating that sat isfied employees perform

better ( Judge et al., 2001). However, a subsequent review that counted the statistica lly sig-

nifica nt findings among early studies concluded that there is ‘m inimal or no relationship