Measurement error in research on financial literacy: How much error is there and how does it influence effect size estimates?

• Published date: 01 June 2022
• Author: Gilles E. Gignac,Elizabeth Ooi
• Date: 01 June 2022
• DOI: http://doi.org/10.1111/joca.12417

COMMENTARY

Measurement error in research on financial

literacy: How much error is there and how does

it influence effect size estimates?

Gilles E. Gignac

1

| Elizabeth Ooi

2

1

School of Psychological Science,

University of Western Australia, Perth,

Western Australia, Australia

2

Business School, University of Western

Australia, Perth, Western Australia,

Australia

Correspondence

Gilles E. Gignac, School of Psychological

Science, University of Western Australia,

35 Stirling Highway, Crawley, WA 6009,

Australia.

Email: gilles.gignac@uwa.edu.au

Abstract

Financial literacy is often measured with only three to

five questions, suggesting challenges with achieving

respectable levels of internal consistency reliability.

Based on our review, we found financial literacy tests

composed of three and five test questions yielded mean

reliability estimates of 0.40 (k=7; N=167,075) and

0.54 (k=8; N=57,937), respectively: values less than

minimally acceptable for even exploratory research.

Based on our more comprehensive review of 52 samples

and a variety of financial literacy tests (3–45 questions),

researchers are recommended to measure financial lit-

eracy with a minimum of 13–15 questions. Finally, we

conclude that the potential impact of financial literacy

on various outcome variables has been underestimated

substantially in many previous investigations, as the

relatively low levels of internal consistency reliability

in the financial literacy test scores attenuated the

obtained effects estimated from the observed scores.

KEYWORDS

effect size, financial capability, financial literacy, reliability

1|INTRODUCTION

Financial literacy may be defined as “…the ability to use knowledge and skills to manage finan-

cial resources effectively for a lifetime of financial well-being”(Knoll & Houts, 2012, p. 383).

Received: 27 June 2021 Accepted: 25 September 2021

DOI: 10.1111/joca.12417

© 2021 American Council on Consumer Interests.

938 J Consum Aff. 2022;56:938–956.

wileyonlinelibrary.com/journal/joca

Research on financial literacy (and the related area of financial capability) has grown substan-

tially over the last 10 years. Consider, for example, that from 2011 to 2020, 8.7 times more finan-

cial literacy/capability papers were published (53 papers vs. 460 papers), based on a Web of

Science search of terms “financial literacy”or “financial capability”(June 14, 2021). By compar-

ison, the same search for papers on the topic of “finance”revealed 2.1 times more papers from

2011 to 2020 (4632 papers vs. 9729 papers). Thus, the area of financial literacy is growing at a

rate more than four times that of the field of finance more broadly, suggesting that it is an

increasingly recognized worthy area of research. However, the rapid growth associated with the

area of financial literacy has perhaps contributed to an oversight in its development: internal

consistency reliability test score estimation and reporting.

In psychometrics, it has been established that the appropriateness of interpreting composite

scores rests upon the observation of a minimally acceptable level of test score reliability; specifi-

cally, internal consistency reliability (Nunnally & Bernstein, 1994). However, across several

published reviews of measures of financial literacy, internal consistency reliability has neither

been considered nor even mentioned (e.g., Huston, 2010; Kimiyaghalam & Safari, 2015;

Ouachani et al., 2020; Remund, 2010). As a rare counter-example, Fernandes et al. (2014,

p. 1868) noted that, “Prior research on measured financial literacy had created measures that

were not subjected to standard psychometric tests to establish unidimensionality, reliability,

and validity…”However, Fernandes et al. (2014) did not estimate the internal consistency reli-

ability of test scores obtained from commonly used financial literacy tests. Furthermore, Fer-

nandes et al. (2014) did not recommend a minimum number of test questions required to

estimate financial literacy reliably: an important consideration, as commonly used measures of

financial literacy include as few as three to five items, that is, the Big Three and the Big Five

(see Hastings et al., 2013). Both the Big Three and the Big Five are used in state-wide surveys

(e.g., National Financial Capability Study; Health and Retirement Study) and many smaller-

scale research investigations (e.g., Davidoff et al., 2017; Goda et al., 2019). Based on their review

of financial literacy measurement, Hastings et al. (2013) concluded that it remains to be deter-

mined whether measuring financial literacy with three or five items is sufficient, as little

research “…has evaluated the relative efficacy of different measurements”(p. 355). Fernandes

et al. (2014) did not address this important question. Consequently, the purpose of this paper

was to introduce, briefly, the concept and estimation of test score reliability, in particular, inter-

nal consistency reliability, as well as its importance in the interpretation of test scores and the

estimation of effect sizes in research. Additionally, we conducted a quantitative review of finan-

cial literacy test score reliability, in order to determine the degree to which financial literacy

tests scores are reliable.

To foreshadow, we conclude that a non-negligible number of published financial literacy

studies are based on test scores with low levels of internal consistency reliability. Furthermore,

as test score reliability limits the magnitude of validity coefficients (e.g., beta-weights; correla-

tions; Nunnally & Bernstein, 1994), much of the research showing financial literacy as a predic-

tor of important outcomes has almost undoubtedly underestimated its potential influence.

1.1 |What is test score reliability?

Test score reliability pertains to the consistency of measurement (Feldt & Brennan, 1989). With-

out consistency in measurement, there is no possibility of validity (Crocker & Algina, 1986).

Thus, all other things equal, greater levels of test score reliability correspond positively to the

COMMENTARY 939