TRENDS: How Internet Search Undermines the Validity of Political Knowledge Measures

Date01 March 2020
Published date01 March 2020
Subject MatterArticles
Political Research Quarterly
2020, Vol. 73(1) 141 –155
© 2019 University of Utah
Article reuse guidelines:
DOI: 10.1177/1065912919882101
Political knowledge is considered the “currency of citi-
zenship” because it helps people process new information
and link their values and interests to their attitudes. A com-
mon way to operationalize this concept is with questions
asking individuals to recall specific facts from memory
(Zaller 1992). Given the centrality of knowledge in stud-
ies of public opinion and political behavior, numerous
debates have occurred over measurement, from the use of
a “Don’t Know” response option (Luskin and Bullock
2011; Miller and Orr 2008; Mondak 2001) and the proper
coding of open-ended questions (Gibson and Caldeira
2015), to item difficulty (Ahler and Goggin 2017) and the
differential functioning of items across demographic
groups (Abrajano 2015). As more surveys are self-admin-
istered over the Internet, a new measurement challenge
has arisen: respondents, who typically complete surveys
at their own pace and without interviewer interaction, can
use search engines to look up information.1
Outside search has been detected in several participant
populations with search rates reaching as high as 41 per-
cent. Evidence of search engine use comes from a variety
of sources, including studies showing higher factual
knowledge scores in the online condition of randomized
mode experiments (Burnett 2016; Clifford and Jerit 2014),
lower scores when respondents are randomly assigned
instructions not to search (Clifford and Jerit 2016; Motta,
Callaghan, and Smith 2016; Vezzoni and Ladini 2017),
correct answers to extremely difficult and obscure open-
ended questions (“catch” questions; Motta, Callaghan,
and Smith 2016), and the outright admission of searching
(Clifford and Jerit 2016; Jensen and Thomsen 2014).
Outside search is common even in high-quality surveys
that have explicitly instructed respondents not to look up
answers. For example, 15 percent of respondents in the
American National Election Studies (ANES) 2018 Pilot
Study looked up the answer to at least one of two catch
questions seconds after being asked not to do so (among
those who received no instructions about search engine
use, 25% looked up at least one answer). Despite the prev-
alence of search engine use, previous work has not defini-
tively answered the question of whether search degrades
the measurement properties of knowledge scales.
We contribute to the literature with a series of studies—
experimental and observational—that employs a range of
criterion measures and varying participant populations.2 In
the first study, we examine the validity of knowledge mea-
sures in experiments where participants are either discour-
aged from seeking outside assistance on factual questions
or told that it is acceptable to do so. In the second study, we
examine search engine use in a national online sample of
882101PRQXXX10.1177/1065912919882101Political Research QuarterlySmith et al.
1U.S. Naval Academy, Annapolis, MD, USA
2University of Houston, Houston TX
3Stony Brook University, Stony Brook, NY
Corresponding Author:
Jennifer Jerit, Social & Behavioral Sciences, Stony Brook University,
7th Floor, Stony Brook, NY 11794, USA.
TRENDS: How Internet Search
Undermines the Validity of Political
Knowledge Measures
Brianna Smith1, Scott Clifford2, and Jennifer Jerit3
Political knowledge is central to understanding citizens’ engagement with politics. Yet, as surveys are increasingly
conducted online, participants’ ability to search the web may undermine the validity of factual knowledge measures.
Recent research shows this search behavior is common, even when respondents are instructed otherwise. However,
we know little about how outside search affects the validity of political knowledge measures. Using a series of
experimental and observational studies, we provide consistent evidence that outside search degrades the validity of
political knowledge measures. Our findings imply that researchers conducting online surveys need to take steps to
discourage and diagnose search engine use.
political knowledge, online surveys, validity

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT