Outside Experts': Expertise and the Counterterrorism Industry in Social Media Content Moderation

• Author: Amre Metwally
• Position: J.D. Candidate, Harvard Law School
• Pages: 471-507

“Outside Experts”: Expertise and the

Counterterrorism Industry in Social Media

Content Moderation

Amre Metwally*

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 471

I. THE PHENOMENON OF EXPERTISE IN TERRORISM STUDIES . . . . . . . . . . . . . 476

II. THE EXPERTISE CONSTELLATIONS IN AND AROUND SILICON VALLEY . . . . 481

A. Inside Experts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 482

B. Outside Experts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485

1. Flashpoint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487

2. Crisp Thinking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488

3. SITE Intelligence Group . . . . . . . . . . . . . . . . . . . . . . . . . 489

4. Middle East Media Research Institute (MEMRI) . . . . . . . 490

III. THE HARMS AND CONCERNS OF EXPERT INFLUENCE ON SOCIAL MEDIA

CONTENT MODERATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 491

A. The Privatization of Public Law Functions in the

Counterterrorism Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . 492

B. Shaping Platforms’ Content Policies and Enforcement . . . . . 496

C. Conditioning Companies to Re-Interpret their Own Terms of

Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499

D. The Problem with Databases and Other Outside Experts’

Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 501

E. Cementing Expert Influence and Faulty Detection Technology

in Social Media Content Regulation . . . . . . . . . . . . . . . . . . . 503

F. Transparency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504

IV. CHECKS AND BALANCES: HOW DO WE CONTROL THIS PROBLEM ? . . . . . 504

CONCLUSION. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 507

INTRODUCTION

Political violence is as old as civilization itself. Wars—local, civil, regional,

and global—are, by their very natures, both political and violent. The

* J.D. Candidate, Harvard Law School. The author was previously a Policy and Enforcement

Manager covering political extremism, counterterrorism, and graphic violence for YouTube. The author

would like to thank the expert advice from Professor Naz Modirzadeh and Cecil Yongo Abungu in

helping to bring this endeavor from thought to finished product. A special thanks as well must be

extended to my fellow participants (Alev Erhan, Shaiba Rather, Carla Yoon, Kathryn Reed, Stephanie

Gullo, Molly Richmond, and Marta Canneri) in Professor Modirzadeh’s International Law writing

group for their incisive feedback and generosity. My deepest gratitude to Adam Silow and the team at

the Journal of National Security Law and Policy for their assistance throughout the editing process. All

errors are strictly my own. © 2022, Amre Metwally.

471

phenomenon of political violence can be reported, categorized, labeled, and dis-

tilled into the distinct acts that comprise it: a killing, a bombing, an execution,

and the list goes on. During the late twentieth and early twenty-first centuries,

particularly from the mid-1900s onward, our lexicon complicated this under-

standing of political violence to introduce a new term: terrorism.

News reports of politically motivated violent acts began advancing this new

term which quickly blossomed. “The year 1972 marked a major transition in the

framing of the media’s treatment of political violence. Events that previously

were covered under the rubrics of assassination, bombing, torture, repression,

massacre, etc., were now classified as ‘terrorism.’ The word (and hence the con-

cept) was catching on.”

1

This fascination with political, violent, and politically

violent drama spread immeasurably with the September 2001 attacks. Since then,

America’s, and more broadly the West’s, “war on terror”

2

Text: President Bush Addresses the Nation, WASH. POST (Sept. 20, 2001), https://perma.cc/D8CF-

TT7V.

spurred the justification

for new invasions,

3

See, e.g., Full Text: Bush’s Speech, GUARDIAN (Mar. 17, 2003, 9:22 PM), https://perma.cc/YFJ8-

M7UR.

creation of new government agencies,

4

and establishment of

new intelligence and information-sharing efforts

5

all designed to “protect” the

United States from its purported enemy, which President George W. Bush

described as “a radical network of terrorists and every government that supports

them.”

6

We know how this story unfolds: invasions into Iraq and Afghanistan, count-

less civilian lives lost, covert torture programs, surveillance programs targeting

Muslims in America, and countries that remain deeply entrenched in political

dysfunction and violence that American intervention either introduced or further

exacerbated.

7

This does not even include the American military’s effort to expand targeted killing programs

through the use of drone aircraft. For more on the drone program, see Christopher J. Fuller, The Origins

of the Drone Program, LAWFARE BLOG (Feb. 18, 2018, 10:00 AM), https://perma.cc/MZS7-8BHA.

While this war on terrorism has already profoundly disrupted the twenty-first

century, there has been at the same time another force unfolding: the technology

industry, and specifically, social media companies. In 2004, the ambitious but

1. JOSEBA ZULAIKA & WILLIAM A. DOUGLASS, TERROR AND TABOO: THE FOLLIES, FABLES, AND

FACES OF TERRORISM 46 (2006).

2.

3.

4. See, e.g., Homeland Security Act of 2002, Pub. L. No. 107–296, 116 Stat. 2135 (2002) (creating

the United States Department of Homeland Security).

5. See, e.g., Uniting and Strengthening America by Providing Appropriate Tools Required to

Intercept and Obstruct Terrorism Act (USA PATRIOT Act) Act of 2001, Pub. L. No. 107–56, 115 Stat.

272 (2001). The PATRIOT Act has ultimately been amended and reauthorized since the initial 2001

law, extending its powers, see, e.g., The USA PATRIOT Improvement and Reauthorization Act of

2005, Pub. L. No. 109-177 (2005). The USA FREEDOM Act ultimately made permanent many

provisions of the PATROIT Act while also limiting the National Security Agency’s bulk collection of

communications material belonging to US citizens. See Uniting and Strengthening America by

Fulfilling Rights and Ensuring Effective Discipline Over Monitoring Act of 2015, Pub. L. No. 114-23,

Stat. 268 (2015).

6. Text: President Bush Addresses the Nation, supra note 2.

7.

472 JOURNAL OF NATIONAL SECURITY LAW & POLICY [Vol. 12:471

naı

¨ve Mark Zuckerberg launched Meta.

8

Our History, ABOUT META, https://perma.cc/4929-N3Z6. The Article uses Facebook’s new name

“Meta” after the company recently changed it.

Steve Chen, Chad Hurley, and Jawed

Karim created YouTube in 2005, now the world’s largest video platform.

9

Laura Fitzpatrick, Brief History YouTube, TIME (May 31, 2010), https://perma.cc/37B8-7ZH4.

And

just one year later, in 2006, Jack Dorsey (who was until recently the company’s

Chief Executive Officer), Evan Williams, and their fellow co-founders changed

the way we communicate, 140 characters at a time.

10

Our Leadership, TWITTER, https://perma.cc/N7NQ-DY8B.

The war on terrorism and technology began to collide in 2014 when social

media intersected with the West’s newest “terrorists” to emerge from Iraq, the

Islamic State in Iraq and Syria (ISIS). As ISIS erupted onto the world stage:

Far from keeping their operation a secret, though, these [ISIS] fighters made sure

everyone knew about it. [.. .] To maximize the chances that the internet’s own

algorithms would propel it to virality, the effort was organized under one telling

hashtag: #AllEyesOnISIS. [.. .] [The hashtag] took on the power of an invisible

artillery bombardment, its thousands of messages spiraling out in front of the

advancing force. Their detonation would sow terror, disunion, and defection.

11

ISIS was not the only group to masterfully wield YouTube, Twitter, and Meta to

broadcast hostage executions, threats, recruitment messages, propaganda, and

battlefield victories.

12

Other actors and organizations flocked to these platforms

as well. For example, Rudaw, a Kurdish news agency, set up a live stream to cap-

ture the fighting; contractors at the US State Department would engage directly

with individuals who seemed likely to join ISIS, and the Iraqi military would

broadcast their wins against the Islamic State.

13

While ISIS may have been one of the first actors to exploit social media’s

potential for its bloody aims in a highly visible manner, they were not the

last. In 2019, Brenton Tarrant live-streamed himself entering multiple mos-

ques in Christchurch, New Zealand, killing 51 worshippers.

14

Christchurch Shootings: 49 Dead in New Zealand Mosque Attacks, BBC (Mar. 15, 2019), https://

www.bbc.com/news/world-asia-47578798.

He teased his

murders on Twitter and then broadcasted the terrorist attack on Meta Live; his

manifesto flourished in YouTube comments and 8chan boards.

15

Kevin Roose, A Mass Murder of, and for, the Internet, N.Y. TIMES (Mar. 15, 2019), https://

perma.cc/PT3M-Y3YQ.

While the

graphic footage of terrorist attacks circulating online is not new, in many ways,

this terrorist attack was “a first — an internet-native mass shooting, conceived and

produced entirely within the irony-soaked discourse of modern extremism.”

16

8.

9.

10.

11. P.W. SINGER & EMERSON T. BROOKING, LIKEWAR: THE WEAPONIZATION OF SOCIAL MEDIA 4-5

(2018).

12. See, e.g., Ahmed Shehabat & Teodor Mitew, Black-boxing the Black Flag: Anonymous Sharing

Platforms and ISIS Content Distribution Tactics, 12 PERSPECTIVES ON TERRORISM 81, 83-85 (2018).

13. SINGER & BROOKING, supra note 11, at 10.

14.

15.

16. Id.

2022] OUTSIDE EXPERTS 473