A Critical Examination of Content Validity Evidence and Personality Testing for Employee Selection

• Author: Christopher R. Milane,David M. Fisher,Sarah Sullivan,Robert P. Tett
• Date: 01 June 2021
• DOI: 10.1177/0091026020935582
• Published date: 01 June 2021
• Subject Matter: Articles

https://doi.org/10.1177/0091026020935582

Public Personnel Management

2021, Vol. 50(2) 232 –257

© The Author(s) 2020

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/0091026020935582

journals.sagepub.com/home/ppm

Article

A Critical Examination of

Content Validity Evidence

and Personality Testing for

Employee Selection

David M. Fisher1, Christopher R. Milane2,

Sarah Sullivan3, and Robert P. Tett1

Abstract

Prominent standards/guidelines concerning test validation provide contradictory

information about whether content-based evidence should be used as a means

of validating personality test inferences for employee selection. This unresolved

discrepancy is problematic considering the prevalence of personality testing, the

importance of gathering sound validity evidence, and the deference given to these

standards/guidelines in contemporary employee selection practice. As a consequence,

test users and practitioners are likely to be reticent or uncertain about gathering

content-based evidence for personality measures, which, in turn, may cause such

evidence to be underutilized when personality testing is of interest. The current

investigation critically examines whether (and how) content validity evidence should

be used for measures of personality in relation to employee selection. The ensuing

discussion, which is especially relevant in highly litigious contexts such as personnel

selection in the public sector, sheds new light on test validation practices.

Keywords

test validation, content validity, personality testing, employee selection

An essential consideration when using any test or measurement tool for employee

selection is gathering and evaluating relevant validity evidence. In the contemporary

employee selection context, validity evidence is generally understood to mean

1The University of Tulsa, OK, USA

2Qualtrics, Provo, UT, USA

3Rice University, Houston, TX, USA

Corresponding Author:

David M. Fisher, Assistant Professor of Psychology, The University of Tulsa, 800 S. Tucker Drive, Tulsa,

OK 74104, USA.

Email: david-fisher@utulsa.edu

935582PPMXXX10.1177/0091026020935582Public Personnel ManagementFisher et al.

research-article2020

Fisher et al. 233

evidence that substantiates inferences made from test scores. Various sources provide

standards and guidelines for gathering validity evidence, including the Uniform

Guidelines on Employee Selection Procedures (Equal Employment Opportunity

Commission, Civil Service Commission, Department of Labor, & Department of

Justice, 1978; hereafter, Uniform Guidelines, 1978), the Principles for the Validation

and Use of Personnel Selection Procedures (Society for Industrial and Organizational

Psychology [SIOP], 2003; hereafter, SIOP Principles, 2003), and Standards for

Educational and Psychological Testing (American Educational Research Association,

American Psychological Association, & National Council on Measurement in

Education, 1999/2014; hereafter, Joint Standards, 1999/2014), as well as the academic

literature (e.g., Aguinis et al., 2001). Having such a variety of sources available is

beneficial, but challenges arise when the various sources provide ambiguous or con-

tradictory information. Such ambiguity can be particularly troublesome in highly liti-

gious contexts, such as the public sector, where adherence to regulations governing

selection is of paramount importance.

The current investigation attempts to shed light on one such area of ambiguity—

whether evidence based on test content should be used as a means of validating per-

sonality test inferences for employee selection. Rothstein and Goffin (2006) noted, “It

has been estimated that personality testing is a $400 million industry in the United

States and it is growing at an average of 10% a year” (Hsu, 2004, p. 156). Given this

reality, it is important to carefully consider appropriate validation procedures for such

measures. However, the various sources mentioned above present conflicting direc-

tions on this issue, specifically in relation to content-based validity evidence. On one

hand, evidence based on test content is one of five potential sources of validity evi-

dence described by the Joint Standards (1999/2014), which is similarly endorsed by

the SIOP Principles (2003). This form of evidence has further been suggested by some

to be particularly relevant to personality tests (e.g., Murphy et al., 2009; O’Neill et al.,

2009), and especially under challenging validation conditions, such as small sample

sizes, test security concerns, or lack of a reliable criterion measure (Landy, 1986; Tan,

2009; Thornton, 2009). On the other hand, the Uniform Guidelines (1978) assert that

“. . . a content strategy is not appropriate for demonstrating the validity of selection

procedures which purport to measure traits or constructs, such as intelligence, apti-

tude, personality, commonsense, judgment, leadership, and spatial ability [emphasis

added]” (Section 14.C.1). Other sources similarly convey reticence toward content

validity for measures of traits or constructs (e.g., Goldstein et al., 1993; Lawshe, 1985;

Wollack, 1976). Thus, there appears to be conflicting guidance on the use of content

validity evidence to support personality measures.

In light of this discrepancy, the current investigation offers a critical examination of

content validity evidence and personality testing for employee selection. Such an

investigation is valuable for several reasons. First, an important consequence of the

inconsistency noted above is that content-based evidence may be overlooked as a

valuable approach to validation when personality testing is of interest. Evidence for

this can be seen in the fact that other approaches such as criterion-related validation

are sometimes viewed as the only option for personality measures (Biddle, 2011).