Measuring the organizational impact of training: The need for greater methodological rigor

• Date: 01 September 2019
• Author: Nicholas Clarke,Yanqing Lai,Ronan Carbery,Valerie Shanahan,Mark N. K. Saunders,Maura Sheehan,Alma McCarthy,Thomas Garavan
• DOI: http://doi.org/10.1002/hrdq.21345
• Published date: 01 September 2019

LITERATURE REVIEW

Measuring the organizational impact of training:

The need for greater methodological rigor

Thomas Garavan

1

| Alma McCarthy

2

| Maura Sheehan

1

|

Yanqing Lai

1

| Mark N. K. Saunders

3

| Nicholas Clarke

4

|

Ronan Carbery

5

| Valerie Shanahan

6

1

Department of International Business,

Edinburgh Napier Business School, Edinburgh,

Scotland, UK

2

Discipline of Management, Cairnes School of

Business and Economics, National University

of Ireland Galway, Galway, Ireland

3

Department of Management, Birmingham

Business School, University of Birmingham

4

Department of Strategy, Leadership and

People, EADA Business School, Barcelona,

Spain

5

Department of Management and Marketing,

University College Cork, Cork, Ireland

6

Global Senior Learning Partner, Squarespace,

Dublin, Ireland

Correspondence

Thomas Garavan, Edinburgh Napier Business

School, Edinburgh, Scotland, UK.

Email: T.Garavan@napier.ac.uk

All authors contributed equally to this article.

We review the methodological rigor of empirical quantitative studies

that have investigated the training and organizational performance

relationship. Through a content analysis of 217 studies published in

quality journals, we demonstrate significant validity threats (in ternal,

external construct, and statistical conclusion validity) that raise ques-

tions about the methodological rigor of the field. Our findings suggest

that the time is appropriate for a renewed methodological endeavor

to understanding the relationship between training and organizational

performance. We make specific recommendations to enhance meth-

odological rigor and generate research findings that will enhance oper-

ationalization of theory, help researchers to make inferences about

causality, and inform the decision-making of Human Resource Devel-

opment (HRD) practitioners.

KEYWORDS

methodological rigor, training and organizational performance,

validity

1|INTRODUCTION

In this paper, we review 40 years of quantitative empirical studies that have investigated the training–organizational

performancerelationship to identifythe methodological features of these studiesand the extent to which theyare sub-

ject to validity threats. Training is an important construct in the HRD and learning and development (L&D) disciplines

(Bell, Tannenbaum, Ford, Noe, & Kraiger, 2017), and numerous industry-based reports document the considerable

investment made by organizations in employee training and development (e.g., Bersin by Deloitte, 2016). In addition,

scholars have argued that training enhances organizational performance, including productivity, innovation, customer

service quality, and financial performance (Aguinis & Kraiger, 2009;Kim & Ployhart, 2014; Noe, Clarke, & Klein, 2014),

yet the evidencebase to make these claims is basedon a preponderance of cross-sectional research designs that shed

DOI: 10.1002/hrdq.21345

© 2019 Wiley Periodicals, Inc.

Human Resource Development Quarterly. 2019;30:291–309. wileyonlinelibrary.com/journal/hrdq 291

little lighton causality. Since 1979,when Miron and McClelland (1979)published the first studyon this relationship,the

past four decadeshave witnessed a sustainedincrease in empirical studiesinvestigating the training–organizationalper-

formance relationship, withmajor growth in publishedstudies since 2010. The extensiveness of pastresearch highlights

the importanceof training in organizations and the need forresearchers to provide practitioners withrobust findings on

the strengthof the relationship, thelinking mechanisms, andthe boundary conditionsexplaining the relationship.

While there are many published reviews and syntheses on the topic of training in organizations (e.g., Bell et al.,

2017; Noe et al., 2014; Salas, Tannenbaum, Kraiger, & Smith-Jentsch, 2012), these reviews have primarily focused on

identifying and reporting key themes and knowledge accumulation on training to date. However, existing reviews sel-

dom engage with the methodological features of studies on the training–organizational performance relationship and

the rigor with whichresearch is undertaken. In contrastto prior reviews, our primary aim in this study is to evaluatethe

methodological characteristics of existing research investigating training and its organizational performance outcomes,

specifically to identify the threats to validity that exist in these studies. Given that the training–organizational perfor-

mance relationship is extensively studied and is centralto the arguments that HRD and L&D specialists make to justify

investmentin training, a major question arisesas to the quality of the evidenceavailable on this relationshipto date.

Three sets of reasons arise for the need to focus on methodological rigor. First, from the perspective of theory, scholars

to date have not always used research designs that reflect the key assumptions of the theories they use to study the relation-

ship. For example, many studies make use of human capital theory (Becker, 1964; Riley, Michael, & Mahoney, 2017) and the

resource-based view (Barney, 1991); however, these theories envisage a long-term contribution of investment in human

resources to organizational performance. Yet the majority of studies use cross-sectional designs and postpredictive designs

(i.e., where respondents provide information on both assessments of current training and their firm’s performanceat the same

time) and therefore do not provide a robust testing of the propositions of the theories used. Wright, Gard ner, Moynihan, and

Allen (2005) describe these designs as postpredictive because they are actually predicting past performance o r performance

up to the point of the survey. Similar arguments are made for studies that utilize social exchange theory (Blau, 1964) and

behavioral theories (Jackson & Schuler, 1995). Therefore, it is reasonable to conclude that some existing studies do not provid e

a robust operationalization of the theoretical foundations of these studies.

Second, from an empirical perspective, two important issues arise. First, there is the problem of contextual validity.

The majority of studies have been conducted in an Anglo-American context (United States, United Kingdom, Canada,

Australia, and New Zealand); therefore, our current understanding of the relationship may not be completely valid given

the emergence of Asia-Pacific, Middle Eastern, and African economies. In addition, the majority of studies focus on pro-

fessional full-time employees, yet the world of employment has changed significantly with the emergence of international

workers and the gig economy. This suggests that the context of the training–organizational performance relationship has

changed in significant ways, thus suggesting a need to understand the complexities of the relationship. A second empirical

reason for analyzing the way in which the training–organizational performance relationship has been investigated con-

cerns the issue of establishing causality. This represents the empirical gold standard of science; however, many existing

studies make use of research designs (typically surveys) that do not enable inferences to be made about causality. Wright

et al. (2005) highlight that survey designs can never ultimately “prove”cause, and many of what are considered well-

designed studies have paid little attention to temporal precedence and/or alternative explanations for the relationship.

This issue has also received prominence in the HRD and training literature. For example, both Sitzmann and Weinhardt

(2018) and Bainbridge, Sanders, Cogin, and Lin (2017) have drawn attention to the needs for greater methodological rigor

in understanding how training and other HRM practices contribute to organizational performance. In the HRD context,

Brown and Latham (2018) highlighted the need for both rigor and relevance in HRD research.

Third, from managerialand HRD practice perspectives,it is important to generate valid insightsand robust research

findings concerning the strength and direction of the relationship between training and organizational performance.

Given that the field of HRD focuses on theinvestigation of L&D processes in workplace settings, it is important that

research findingswithin the field should inform practice in these settings.Thus, an important motivation for this study

speaks to recent debates concerning the role of research in generating evidence that is of value in the real world

(Brown & Latham, 2018; Gubbins, Harney, van de Werff, & Rousseau, 2018). This discussion suggests that academic

292 GARAVAN ET AL.