Revisiting trainee reactions: A multilevel analysis of the nomological network

• Author: Minjung Kim,Sanghamitra Chaudhuri,John Lavelle,Sehoon Kim,Soyoun Park
• Date: 01 June 2020
• Published date: 01 June 2020
• DOI: http://doi.org/10.1002/hrdq.21383

QUANTITATIVE STUDY

Revisiting trainee reactions: A multilevel analysis

of the nomological network

Sehoon Kim

1

| Soyoun Park

2

| John Lavelle

1

| Minjung Kim

1

|

Sanghamitra Chaudhuri

3

1

University of Minnesota, Minneapolis,

Minnesota

2

Anyang University, Anyang, Republic of

Korea

3

Metropolitan State University, Minneapolis,

Minnesota

Correspondence

Soyoun Park, Anyang University, Anyang,

Gyeonggi-do 430-714, Republic of Korea.

Email: soyounpark2020@gmail.com

Abstract

Surveying participants' reactions remains the most popular

approach to evaluate training programs. However, it is

unclear how reactions are related to training contexts and

outcomes. Using a multilevel analysis, this study examined

the structural relationships among individual/situational

characteristics, trainee reactions, and training outcomes in

mandatory safety and health training programs. We ana-

lyzed data from 545 participants of 29 face-to-face training

classes who completed three sequential surveys. The results

revealed that pre-training motivation and the classroom set-

ting are associated with reactions, and reactions are associ-

ated with participatory behavior, cognitive learning, and

perceived transfer. Trainee reactions significantly mediated

the relationships of pre-training motivation and the class-

room setting with training outcomes. The significance of

this study is identifying the roles of trainee reactions in

training, applying a multilevel analysis in training research.

KEYWORDS

affective reactions, learning, participation, reactions, situational

characteristics, trainee characteristics, transfer, utility reaction

1|INTRODUCTION

Formal training plays a critical role in improving the knowledge and skills that employees need for their jobs

(Becker & Bish, 2017; Bednall & Sanders, 2017). Although informal approaches to learning have recently drawn

increased scholarly attention, formal training is still heavily used and has shown evidence of positive effects on many

situations (Manuti, Pastore, Scardigno, Giancaspro, & Morciano, 2015; Ricci, Chiesi, Bisio, Panari, & Pelosi, 2016).

DOI: 10.1002/hrdq.21383

© 2019 Wiley Periodicals, Inc.

Human Resource Development Quarterly. 2020;31:173–191. wileyonlinelibrary.com/journal/hrdq 173

In the process of formal training, surveying participants' reactions remains the most popular evaluation approach in

practice (Giangreco, Sebastiano, & Peccei, 2009; Harman, Ellington, Surface, & Thompson, 2015; Kucherov & Man-

okhina, 2017; Morgan & Casper, 2000). A reason for the common use of trainee reactions in practice seems to be

because it is easy and fast for trainers to collect reaction-level data from participants. In addition, both trainers and

organizations may believe that (a) these data are a proxy for outcome measures and (b) any data must be better than

no data at all. Some scholars also believe that reaction data are useful for predicting subsequent outcomes such as

learning and transfer of what was learned to the job (Clement, 1982; Hughes et al., 2016; Madera, Steele, &

Beier, 2011).

The majority of the research, however, has highlighted the limitations of and overreliance on trainee reactions in

evaluation (Holton, 1996; Sitzmann & Weinhardt, 2019; Swanson, 2005), which has in turn called into question its

validity for evaluating training programs. In fact, the current literature does not provide clear evidence on trainee

reactions in training because the theoretical scholarship and empirical research have generated mixed results. Some

studies have found that reactions have no effect on either learning or transfer (Kraiger, 2002; Mathieu,

Tannenbaum, & Salas, 1992). In contrast, work by Diamantidis and Chatzoglou (2012) indicated that trainee reactions

are related to both learning and transfer. In addition, Blume, Ford, Baldwin, and Huang's (2010) meta-analysis found

that overall reactions do not relate to transfer, but Tan, Hall, and Boyce (2003) found a negative relationship.

These mixed findings may be attributed to several limitations in the existing literature. First, previous studies on

training reactions tended to employ simple descriptive and univariate analyses. These analyses can be helpful in

understanding the basic nature of the trainee reactions and how reactions might be related in time to either the char-

acteristics of a specific training event and/or its subsequent outcomes. However, the descriptive and univariate

approaches are limited because training is a complex process with multiple variables that influence the relationship

between the training event and the constellation of possible outcomes. As such, a more complex and nuanced

approach is necessary, such as analyzing the nomological network in a structural model.

Another limitation is that scholars have used different constructs of reactions in post-training surveys. It is critical

for practitioners and scholars to distinguish between affective reactions (e.g., satisfaction and pleasure) and utility

reactions (e.g., perceived application of the training) though both are important to include because they occur simul-

taneously and influence training outcomes (Blume et al., 2010; Sitzmann, Brown, Casper, Ely, & Zimmerman, 2008).

However, reaction evaluations traditionally ask only about participants' affective reactions to particular elements of

the training (e.g., content, delivery method, instructor, and environment; Lee & Pershing, 2002).

Research also provides insufficient information on how training contexts influence reactions. Although

psychology-based studies have found relationships between individual traits and learning experiences (Ainley, Hidi, &

Berndorff, 2002; Weissbein, Huang, Ford, & Schmidt, 2011), there is little research on how situational characteristics

influence training. Examples of these situational characteristics include the physical environment in which the train-

ing takes place and administrative support for the implementation of the training itself.

Finally, previous training studies have mostly ignored class-level variance when the data were collected from

multiple training classes (Mathieu & Tesluk, 2010). Given that training classes are likely to have different instruc-

tional, administrative, and environmental contexts that influence trainee experiences and training outcomes, training

studies should address the need for multilevel analysis (Harman et al., 2015).

This study aims to examine the relationships between individual and situational characteristics, trainee reactions,

and training outcomes using a multilevel structural approach. Specifically, we focus on (a) the main effects of individ-

ual and situational characteristics on trainee reactions, (b) the main effects of trainee reactions on training outcomes,

and (c) the mediation effects of trainee reactions on the relationship of individual and situational characteristics with

training outcomes. By testing trainee reactions' relationships, the significance of this study for human resource devel-

opment (HRD) includes guiding practitioners to incorporate practical and effective uses of trainee reactions to train-

ing and convincing researchers of the need to examine reactions in training evaluations. The results will inform HRD

theories and help research address issues related to training employees, which can ultimately lead to more effective

training programs.

174 KIM ET AL.