Political Attacks in 280 Characters or Less: A New Tool for the Automated Classification of Campaign Negativity on Social Media

• Published date: 01 May 2022
• DOI: 10.1177/1532673X211055676
• Date: 01 May 2022
• Author: Vladislav Petkevic,Alessandro Nai
• Subject Matter: Articles

Article

American Politics Research

2022, Vol. 50(3) 279–302

© The Author(s) 2021

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/1532673X211055676

journals.sagepub.com/home/apr

Political Attacks in 280 Characters or Less: A

New Tool for the Automated Classification of

Campaign Negativity on Social Media

Vladislav Petkevic

1

,and Alessandro Nai

2



Abstract

Negativity in election campaign matters. To what extent can the content of social media posts provide a reliable indicator of

candidates’campaign negativity? We introduce and critically assess an automated classification procedure that we trained to

annotate more than 16,000 tweets of candidates competing in the 2018 Senate Midterms. The algorithm is able to identify the

presence of political attacks (both in general, and specifically for character and policy attacks) and incivility. Due to the nov el

nature of the instrument, the article discusses the external and convergent validity of these measures. Results suggest that

automated classifications are able to provide reliable measurements of campaign negativity. Triangulations with independent

data show that our automatic classification is strongly associated with the experts’perceptions of the candidates ’campaign.

Furthermore, variations in our measures of negativity can be explained by theoretically relevant factors at the candidate and

context levels (e.g., incumbency status and candidate gender); theoretically meaningful trends are also found when replicating

the analysis using tweets for the 2020 Senate election, coded using the automated classifier developed for 2018. The implications

of such results for the automated coding of campaign negativity in social media are discussed.

Keywords

negative campaigning, US Midterms, machine learning, neural networks, incivility

Introduction

Modern politics is a hard-fought business. The public is

increasingly hostile toward those they consider as their rivals

(Iyengar et al., 2012;Iyengar & Westwood, 2015), antago-

nistic and aggressive political figures are on the rise across the

globe (Nai & Martinez i Coma, 2019a), and attacks seem at

times the very essence of election campaigns (Ansolabehere

et al., 1994;Lau & Pomper, 2004). To be sure, negativity in

politics matters. Negative information, when compared to

equivalent “positive”information, is more likely to be seen,

processed, and remembered (e.g., Rozin & Royzman, 2001).

Also because of that, some scholars show that negative

messages can convey important and useful information to the

voters, promote issue knowledge, cue the voters that the

election is salient, and thus worth the emotional and cognitive

investment, and ultimately stimulate the interest of the public

(Finkel & Geer, 1998;Martin, 2004;Geer, 2006). On the

other hand, however, strong evidence suggests that negative

campaigning can be a detrimental force in modern democ-

racies. Negative and harsh campaigns can reduce turnout and

political mobilization, depress civic attitudes such as political

efficacy and trust, foster apathy, and generally produce a

“gloomier”public mood (Ansolabehere et al., 1994;Thorson

et al., 2000;Yoon et al., 2005). On top of this, a case can be

made that decreased trust in the political game and increased

political cynicism are likely to reinforce the consolidation of

antagonistic and disruptive movements, which often “feed”

off the public discontent. Whether a positive or detrimen tal

force, few would contest that negativity is a key component of

contemporary electoral democracies.

In recent years, the dynamics of electoral campaigning

have been reshaped by the emergence of social media

(Gainous & Wagner, 2014;Graham et al., 2016;Straus et al.,

2013). Online communication, especially via social media,

allows political actors to “cut the middlemen”—for instance,

journalistic gatekeeping—and communicate directly with

their audience, in what is often referred to as “one-step flow of

1

Faculty of Social and Behavioral Sciences, University of Amsterdam,

Amsterdam, Netherlands

2

Amsterdam School of Communication Research (ASCoR), University of

Amsterdam, Amsterdam, Netherlands

Corresponding Author:

Vladislav Petkevic, Faculty of Social and Behavioral Sciences, University of

Amsterdam, Nieuwe Achtergracht 166, 1018 WV, Amsterdam, Netherlands.

Email: v.petkevich@uva.nl

communication”(Bennett & Manheim, 2006). Such facili-

tated access to the people is one of the reasons why online

communication is particularly favored by populists (Engesser

et al., 2017). In recent years, several studies have assessed the

presence of negativity in social media (e.g., Auter & Fine,

2016;Ceron & d’Adda, 2016;Evans et al., 2014;Evans &

Clark, 2016;Gainous & Wagner, 2014;Gross & Johnson,

2016). Broadly speaking, these studies find confirmation that

the main trends of strategic campaigning found for traditional

techniques—for instance, that challengers tend to attack more

than incumbents (Gainous & Wagner, 2014)—are also found

when looking at campaigning on social media. Those existing

studies tended to rely on manual coding of social media posts.

For instance, in their analysis of the drivers of negativity of

Facebook during the 2010 Midterms, Auter and Fine (2016)

manually coded more than 14,000 posts. Similarly, Ceron and

d’Adda (2016) hand-coded more than 15,000 Tweets pub-

lished by competing candidates prior to the 2013 Italian

general election. Recent advances of machine learning ap-

proaches have made it increasingly affordable to dive into

very large amounts of data, which were inaccessible—or

required time-intensive coding efforts—up to recently due to

insufficient computational power and the preference for

manual coding. In this article, we expand the growing lit-

erature on automated classification of textual data within the

context of political communication. We introduce a neural

network classifier that we trained to automatically annotate

the tweets of candidates competing during the 2018 US

Senate Midterms elections, in terms of the presence of po-

litical attacks. The algorithm was run on approximately

16,000 tweets, posted by 63 candidates for the period be-

tween September 1

st

and November 6

th

, 2018 (the day of the

election). After presenting the results of the classification, the

article will test the external and convergent validity of the

measure; more specifically, we will check whether the results

make sense in terms of factors that can be theoretically ex-

pected to drive the presence of negativity in the candidates’

tweets (e.g., the incumbency status of the candidate), and by

triangulating the measure with an independent dataset about

the content of candidates’campaigns in the 2018 Midterms,

as assessed by expert surveys (Nai & Maier, 2020).

The reason for studying the 2018 midterm elections was

two-fold. Firstly, from a conceptual standpoint, the US senate

elections provide a unique opportunity to study a series of

extremely similar elections with a reduced number of com-

petitors, happening simultaneously within the same broad

societal, cultural and, ultimately, political context (Lau &

Pomper, 2001,2004)—while, at the same time, being able to

control for the most important differences at the contextual

level (e.g., how close the race was). Yet, even if driven by

state-level dynamics, Senate Midterms elections all partici-

pate to the broader national context and political dynamics.

The 2018 Midterms were not an exception in this sense, and

the results in each state had fundamental national implications

in terms of, for example, the control of the upper house (so

central to the recent dynamics of presidential impeachment of

Donald J. Trump). In other terms, the Senate Midterms are an

ideal research setting, allowing all the benefits of variation—

both at the candidate and context levels—while keeping most

of the broad cultural and political dynamics, assumed to be

shared across all state-level elections at bay, so to speak.

Indeed, especially compared with Presidential elections,

Senate elections can be seen as “methodologically superior”

for the study of campaign dynamics (Lau & Pomper, 2004,p.

6). Secondly, the 2018 Midterm Senate elections provide us

with the unique opportunity to test the convergent validity of

our data by comparing it with other, independent data about

the same elections and the same dynamics (i.e., how negative

the candidates in the Midterms went against each other).

More specifically, we will triangulate the tone of the can-

didates’campaign on Twitter with expert ratings provided by

independent scholars (Nai & Maier, 2020).

The rest of this article proceeds as follows. In section 2,we

describe the empirical procedure employed to develop the

algorithm for the automated coding of the negativity in

tweets. Section 3 then presents three tests. First, we test the

convergent validity of the algorithm, by comparing it with the

measure of negativity from independent data using expert

surveys. Second, we test its external validity by checking our

measurement against some trends that can be theoretically

expected (i.e., the fact that challengers should be expected to

be more likely to go negative, or that female candidates tend

to use gentler campaigns. Finally, third, we investigate

whether applying the coding algorithm to a different set of

data—the campaign on Twitter during the 2020 Senate

election—yields results that are also theoretically valid. As

we will see, our algorithm scores well in both external and

convergent validity, suggesting that the automated coding of

social media posts is an effective alternative to standard

measurement of campaign content. The last section concludes

the discussion and glimpses over the directions of future

research.

Supporting materials for this article are available at the

following Open Science Foundation repository: https://osf.io/

up826/. The repository includes (i) the Jupyter Notebook

(Python) file with the code that was used to pre-process the

raw data, build the classifier, and annotate the whole dataset,

(ii) the annotated dataset of all tweets, (iii) the archive with

the text of all the collected tweets, and (iv) an excel file with

the reliability assessment of the initial sample of tweets coded

(see below).

Measuring Negative Campaigning in Tweets

Data and Procedure

During the 2018 Midterms, 33 Class 1 Senate seats were up

for grabs (plus additional special elections in Minnesota and

Mississippi to fill vacancies, but which we will not analyze

here); Democrats were holding 26 of these seats, and

280 American Politics Research 50(3)