TRENDS: How Internet Search Undermines the Validity of Political Knowledge Measures

• Date: 01 March 2020
• Published date: 01 March 2020
• DOI: 10.1177/1065912919882101
• Subject Matter: Articles

https://doi.org/10.1177/1065912919882101

Political Research Quarterly

2020, Vol. 73(1) 141 –155

© 2019 University of Utah

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/1065912919882101

journals.sagepub.com/home/prq

Article

Political knowledge is considered the “currency of citi-

zenship” because it helps people process new information

and link their values and interests to their attitudes. A com-

mon way to operationalize this concept is with questions

asking individuals to recall specific facts from memory

(Zaller 1992). Given the centrality of knowledge in stud-

ies of public opinion and political behavior, numerous

debates have occurred over measurement, from the use of

a “Don’t Know” response option (Luskin and Bullock

2011; Miller and Orr 2008; Mondak 2001) and the proper

coding of open-ended questions (Gibson and Caldeira

2015), to item difficulty (Ahler and Goggin 2017) and the

differential functioning of items across demographic

groups (Abrajano 2015). As more surveys are self-admin-

istered over the Internet, a new measurement challenge

has arisen: respondents, who typically complete surveys

at their own pace and without interviewer interaction, can

use search engines to look up information.1

Outside search has been detected in several participant

populations with search rates reaching as high as 41 per-

cent. Evidence of search engine use comes from a variety

of sources, including studies showing higher factual

knowledge scores in the online condition of randomized

mode experiments (Burnett 2016; Clifford and Jerit 2014),

lower scores when respondents are randomly assigned

instructions not to search (Clifford and Jerit 2016; Motta,

Callaghan, and Smith 2016; Vezzoni and Ladini 2017),

correct answers to extremely difficult and obscure open-

ended questions (“catch” questions; Motta, Callaghan,

and Smith 2016), and the outright admission of searching

(Clifford and Jerit 2016; Jensen and Thomsen 2014).

Outside search is common even in high-quality surveys

that have explicitly instructed respondents not to look up

answers. For example, 15 percent of respondents in the

American National Election Studies (ANES) 2018 Pilot

Study looked up the answer to at least one of two catch

questions seconds after being asked not to do so (among

those who received no instructions about search engine

use, 25% looked up at least one answer). Despite the prev-

alence of search engine use, previous work has not defini-

tively answered the question of whether search degrades

the measurement properties of knowledge scales.

We contribute to the literature with a series of studies—

experimental and observational—that employs a range of

criterion measures and varying participant populations.2 In

the first study, we examine the validity of knowledge mea-

sures in experiments where participants are either discour-

aged from seeking outside assistance on factual questions

or told that it is acceptable to do so. In the second study, we

examine search engine use in a national online sample of

882101PRQXXX10.1177/1065912919882101Political Research QuarterlySmith et al.

research-article2019

1U.S. Naval Academy, Annapolis, MD, USA

2University of Houston, Houston TX

3Stony Brook University, Stony Brook, NY

Corresponding Author:

Jennifer Jerit, Social & Behavioral Sciences, Stony Brook University,

7th Floor, Stony Brook, NY 11794, USA.

Email: jennifer.jerit@gmail.com

TRENDS: How Internet Search

Undermines the Validity of Political

Knowledge Measures

Brianna Smith1, Scott Clifford2, and Jennifer Jerit3

Abstract

Political knowledge is central to understanding citizens’ engagement with politics. Yet, as surveys are increasingly

conducted online, participants’ ability to search the web may undermine the validity of factual knowledge measures.

Recent research shows this search behavior is common, even when respondents are instructed otherwise. However,

we know little about how outside search affects the validity of political knowledge measures. Using a series of

experimental and observational studies, we provide consistent evidence that outside search degrades the validity of

political knowledge measures. Our findings imply that researchers conducting online surveys need to take steps to

discourage and diagnose search engine use.

Keywords

political knowledge, online surveys, validity