What do we really know about “don't know”? Re‐assessing the measurement of financial knowledge

• Published date: 01 October 2023
• Author: Melissa J. Wilmarth,Kyoung Tae Kim,Tae‐Young Pak
• Date: 01 October 2023
• DOI: http://doi.org/10.1111/joca.12563

RESEARCH ARTICLE

What do we really know about “don't know”?

Re-assessing the measurement of financial

knowledge

Melissa J. Wilmarth

1

|Kyoung Tae Kim

1

|Tae-Young Pak

2

1

Department of Consumer Sciences,

University of Alabama, Tuscaloosa,

Alabama, USA

2

Department of Consumer Science and

Convergence Program for Social

Innovation, Sungkyunkwan University,

Seoul, South Korea

Correspondence

Tae-Young Pak, Department of

Consumer Science and Convergence

Program for Social Innovation,

Sungkyunkwan University, Seoul,

South Korea.

Email: typak@skku.edu

Funding information

National Endowment for Financial

Education

Abstract

Survey protocols measuring financial knowledge typ-

ically offer a don't know (DK) response option along

with factual statements about personal finance. The

potential problem of this practice is that a DK

response could capture something other than finan-

cial knowledge and mislead empirical research on

the association between financial knowledge and

behavioral outcomes. In this study, we examined

whether the current scales are contaminated by sys-

temic personality effects and how reduced validity

influences analytical modeling of the knowledge

effect. Two studies with different national datasets

were conducted in this investigation. Study 1 found

that personality types and emotions are partially cor-

related with the propensity to give a DK response.

Study 2 showed that controlling for DK response

options alters the association between financial

knowledgeandbehaviorsinregressionanalyses.Our

findings suggest that a DK response reduces con-

struct validity of the financial knowledge score.

KEYWORDS

construct validity, don't know, financial knowledge, financial

literacy, measurement

Received: 30 June 2022Revised: 26 August 2023Accepted: 2 October 2023

DOI: 10.1111/joca.12563

© 2023 American Council on Consumer Interests.

J Consum Aff. 2023;57:1623–1649. wileyonlinelibrary.com/journal/joca 1623

1|BACKGROUND

Research has shown that understanding money management is critically important to make

informed personal finance decisions and avoid costly financial mistakes (Lusardi &

Mitchell, 2014).However, one of the pivotal findings in consumer behavior researchhas been that

Americans know little about personal finance (Hogarth & Hilgert, 2002; Lusardi, 2019;Lusardi&

Mitchell, 2007; Urban & Valdes, 2022). Despite a growing scholarly interest infinancial knowl-

edge, relatively little is known about the validity of survey protocols for measuring financial

knowledge. The standard survey procedure for financial knowledge presents survey participants

with a series of definitive statements about personal finance topics and asks participants to select

the correct answer. Common response formats include true/false or three- to four-category multi-

ple choice items as well as a don't know (DK) option. The summary score of financial knowledge

is typically defined as the count or factor score of correct answers given by each respondent. Con-

sequently, the scoring procedure implicitly assumes that an incorrect and a DK response indicate

the same knowledge state—an absence of knowledge (Mondak, 2000; Mondak & Davis, 2001).

We contend that this approach may distort our understanding of the depth of financial illiteracy

and its effect on behavioral outcomes in empirical analyses.

Researchers in educational testing have shown that a knowledge battery with a DK option

invites systematic error associated with the propensity to guess (Cronbach, 1946; Mehrens &

Lehmann, 1978). Under the current survey protocol to merge incorrect answers and DK

responses, two respondents with the same level of knowledge in personal finance may receive

different scores to the extent that one person is more or less likely to select a DK response

option. A respondent who provides a substantive judgment enjoys the random chance of choos-

ing the correct answer, while the one who chooses a DK response option forgoes this opportu-

nity and obtains a lower score. While this tendency would have minimal impact on the

measurement of financial knowledge if all respondents exhibit a similar propensity to select a

DK response option, unfortunately, this is not the case. According to literature in educational

testing, the propensity to give a DK response varies based on respondents' personality types,

such as self-confidence, competitiveness, and propensity to take risks (Cronbach, 1946;

Hirschfeld et al., 1995; Sherriffs & Boomer, 1954). Research on political knowledge demon-

strated that including DK response options in knowledge surveys opens the possibility for per-

sonality types to contaminate the knowledge score (Mondak, 2000). This suggests that the

current financial knowledge tests may tap two distinct constructs, that is, knowledge and pro-

pensity to guess, and therefore lack construct validity as a survey battery.

One solution that researchers have posited is considering the selection of a DK response

option as akin to missing data. Beatty and Herrmann (2002) identified that when a respondent

is willing, but unable to respond, the data could possibly be accounted as missing at random.

However, this does not always apply in relation to knowledge tests, as a selection of the DK

response option could truly be related to the question itself. The choice to include or not include

a DK response option in surveys is a difficult one, especially when it comes to the utilization

and treatment of the collected data in empirical analyses. Not all DK responses are equal as

question topics and level of difficulty play a role in a respondent's selection of a DK response

option; see de Leeuw et al. (2016) for a review of data nonresponse prevention and treatment in

survey data collection.

The validity of the knowledge battery has received limited attention in the financial knowl-

edge and personal finance literature. Knoll and Houts (2012) reviewed existing financial knowl-

edge questions and used an item-response theory, leading to a 20-item financial knowledge

1624 WILMARTH ET AL.