On the Internet, No One Knows You’re an Activist: Patterns of Participation and Response in an Online, Opt-in Survey Panel

• Published date: 01 December 2024
• DOI: http://doi.org/10.1177/10659129241268818
• Author: Daniel J. Hopkins,Tori Gorton
• Date: 01 December 2024

Article

Political Research Quarterly

2024, Vol. 77(4) 1397–1414

© The Author(s) 2024

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/10659129241268818

journals.sagepub.com/home/prq

On the Internet, No One Knows You’re an

Activist: Patterns of Participation and

Response in an Online, Opt-in Survey Panel

Daniel J. Hopkins

1

and Tori Gorton

2

Abstract

Recentyearshaveseenadramaticincreaseinsurveysofonlinerespondents who participate absent random selection. Using

an uncompensated, opt-in panel of 11,000 Pennsylvanians (2020–2022), we benchmark self-reports against vote histories,

campaign contributions, election returns, and Census data. Ours is among the f‌irst such benchmarking exercises in more than

a decade, a period when online survey research has transformed. It is also among the f‌irst analyzing a longitudinal panel. Our

panelists are approximately balanced on partisanship; attrition is limited. Notably, respondents are very politically engaged,

with high rates of turnout, campaign contributions, and rally attendance. Ethnic/racial minorities are under-represented.

Political attitudes show high over-time stability, and an experiment uncovers few partisan differences in response rates. Such

survey samples are cost-effective ways to track the views of highly engaged citizens over time, but less engaged citizens prove

hard to recruit, even with supplemental efforts. Survey participation is itself a political behavior.

Keywords

online panels, opt-in surveys, survey experiments, non-response, activists, political engagement

In recent decades, the use of online, opt-in surveys to study

Americans’political attitudes has exploded. To date, survey

experiments with opt-in samples have shown noteworthy

external validity, whether respondents are recruited via online

markets or empaneled directly to take surveys (Berinsky et al.

2012;Coppock et al. 2018;Mullinix et al. 2015).

1

Still, there are outstanding questions about non-

probability samples (Cornesse et al. 2020;Eyal et al.

2021;Krupnikov et al. 2021). In contrast to earlier prac-

tice, researchers using these samples are typically not in-

volved in sample recruitment, and so aren’t as familiar with

the procedures that lead respondents to be included in their

data (Enns and Rothschild 2023). The fraction of the target

population available for surveys is thus unclear but is likely to

be small (Stewart et al. 2015). In particular, online, opt-in

surveys risk over-representing the highly politically engaged

(Cavari and Freedman 2022;JackmanandSpahn2021).

We address these questions by detailing a multi-year

panel surveywhich we directed with the surveyf‌irm Civiqs

serving as recruiter and interviewer.

2

Certainly, previous

studies have sought to understand online, opt-in survey

respondents by benchmarking them relative to other pop-

ulations, principally making comparisons to high-quality

governmental surveys (e.g., Ansolabehere and Schaffner

2014;Dutwin and Buskirk 2017;Keeter 2018;Kennedy

et al. 2016;MacInnis et al. 2018;Malhotra and Krosnick

2007;Rivers and Bailey 2009;Vavreck andRivers 2008;

Yeager et al. 2011).Yet amongthose benchmarkingstudies,

the most recent data in a peer-reviewed article comes from

2014, which was a decade ago and prior to the signif‌icant

polling errors in the U.S. and U.K. (Clinton et al. 2022;

Prosser and Mellon 2018). The populationof respondentsto

online, non-probability samples can change signif‌icantly in

a few months (Aronow et al. 2020), to say nothing of a

decade. Many of the main providers of online, opt-in

samples for contemporary political science did not exist a

decade ago (Coppock and McClellan 2019).

Even beyond its recent vintage, the panel we report has

several features which make it an especially valuable tool

1

University of Pennsylvania, Philadelphia, PA, USA

2

Independent Researcher, Philadelphia, PA, USA

Corresponding Author:

Daniel J. Hopkins, Julie and Martin Franklin Presidential Professor,

Department of Political Science, University of Pennsylvania, Perelman

Center for Political Science and Economics, 133 S. 36th Street,

Philadelphia, PA 19104, USA.

Email: danhop@sas.upenn.edu

through which to assess the opportunities and limits of

samplesrecruited onlineto take uncompensated surveys. As

Appendix Table 8 illustrates, Pennsylvania is broadly

representative of the U.S. overall. Recruiting in thislarge

American state enabled us to compare our survey-based

estimates with external benchmarks, including public rec-

ords of vote history and campaign contributions alongside

election returns, Census data, and newspaper reports. In

fact, this studyis among the only to use off‌icial, individual-

level voter records as an externally validated benchmark

when evaluating the bias in online, opt-in survey data —and

the f‌irst to our knowledge in more than a decade (see also

Rivers and Bailey 2009;Vavreck and Rivers 2008).

We also contribute by evaluating the properties of not

just a one-time cross-section but a multi-wave panel (see

also Vavreck and Rivers 2008). Doing so is especially

important given the increased use of panels to measure the

outcomes of f‌ield experiments (Broockman et al. 2017).

By assessing who participates in such panels, we will be

better positioned to understand the populations to whom

such f‌ield experiments generalize.

Via online advertisements on Facebook, Google, and

Instagram, we recruited sizable numbers of Democrats and

Republicansto take brief surveys, with high re-interview

rates that facilitate over-time study. These respondents were

recruited via Civiqs’usual recruitment channels and were

combined with pre-existing members of Civiqs’panel who

were Pennsylvania residents—48% of our full panel was

newly recruited for this project, with the remaining 52%

coming from Civiqs’existing panelists. However, our

sample is very politically engaged: of those we were able to

matchto voter f‌iles, 87% voted in the 20 18 midterm election

and 72% in the 2020 primary. The actual f‌igures in

Pennsylvania were signif‌icantly lower, at 58% and 34%,

respectively. We also f‌ind levels of campaign contributions

and Trump rally attendance far higher than among Penn-

sylvanian adults overall. Our respondents’vote choice

preferences are very stable overa campaign, even relativeto

a national, population-based panel. Deliberate efforts to

reduce such biases by targeting lower-engagement Penn-

sylvanians yielded few additional respondents.

These results reinforce an expressive account of survey

response, in which those who are most motivated to

participate are highly engaged partisans (see also Graham

and Huber 2021). However, in contrast to Gelman et al.

(2016), such motivations vary little over time among this

sample. An experiment before and after the November

2021 election uncovers no evidence that individuals are

motivated to respond to our survey by events that are

positive or negative for their party.

Our results also corroborate prior f‌indings (Hillygus

et al. 2014;Kennedy et al. 2016;MacInnis et al. 2018):

racial/ethnic minorities are under-represented in our panel

relative to their share of the population, as are younger

voters. Comparisons with benchmarks from the Current

Population Survey (CPS) illustrate that lower income Black

registered voters and Black or Latino registered voters

without college degrees are particularly under-represented.

Researcher involvement in the design and recruiting of

panelists provides procedural transparency and opportu-

nities for experimentation.

3

Unlike some similar online,

non-probability surveys, our panel does not offer re-

spondents material incentives (see MacInnis et al. 2018).

Recruiting respondents to take free, short surveys avoids

the need to solicit personal information for payment, and

so has the potential to include some respondents who

would not otherwise participate while reducing the in-

centive for disruptive bots (Aronow et al. 2020).

While many contemporary online, opt-in surveys provide

compensation (e.g., Ansolabehere et al. 2023), it is none-

theless key to understand samples motivated by

non-monetary incentives, in part because such expressive

incentives are likely to play some role even in surveys in

which respondents are modestly compensated. That said,

recent research has commonly employed uncompensated

surveys, at times because certain research designs make

compensating otherwise anonymous survey respondents

diff‌icult and/or because of the need to maintain anonymity

(e.g., Carpenter 2012;Hopkins et al. 2024;Hopkins and

Gorton 2024;Medeiros and Gravelle 2023;Weiss 2021;

Williams, Gravelle and Klar 2022;Zhang et al. 2018).

(Surveys administered through other modes, such as tele-

phone surveys and exit polls, are also commonly

uncompensated—e.g., Croco et al. 2019;Tyler and Iyengar

2023;Sambanis, Simonson, and Yaylacı2023—so insights

from this study may also apply to those survey modes.)

We def‌ine activists as those who engage in political

activities beyond voting (Hopkins and Noel 2022). In re-

cent years, such activists have arguably been understudied

relative to both political elites and the mass public (Blum

2020;Han 2014;Parker and Barreto 2014). These results

clarify that online, opt-in samples can effectively survey

such highly engaged partisans repeatedly at relatively low

cost.

4

When coupled with prior results on treatment effect

stability (Berinsky et al. 2012;Coppock et al. 2018;

Mullinix et al. 2015), they indicate a meaningful role for

free, online, opt-in samples in contemporary research

(Krupnikovet al. 2021). This approach thusopens up many

opportunitiesfor studying activists from both parties.

However,these results also clarifythat our ability to sample

lower-engagement voters is quite limited—andthatincases

where researchers are interested in that side of the en-

gagement divide (Krupnikov and Ryan 2022), or in Black

or Latino citizenswithout college degrees,they will need to

employ alternatives. Moreover, free surveys may dispro-

portionately attract those with expressive motivations—

such as strong, highly engaged partisans and activists—

and may also have lower response rates (Yan et al. 2018).

1398 Political Research Quarterly 77(4)