Reassessing the Role of Theory and Machine Learning in Forecasting Civil Conflict

• Author: Andreas Beger,Michael D. Ward,Richard K. Morgan
• Published date: 01 August 2021
• Date: 01 August 2021
• DOI: http://doi.org/10.1177/0022002720982358
• Subject Matter: Author Exchange

Author Exchange

Reassessing the Role

of Theory and Machine

Learning in Forecasting

Civil Conflict

Andreas Beger

1

, Richard K. Morgan

2

,

and Michael D. Ward

1,3,4

Abstract

We examine the research protocols in Blair and Sambanis’ recent article on fore-

casting civil wars, where they argue that their theory-based model can predict civil

war onsets better than several atheoretical alternatives or a model with country-

structural factors. We find that there are several important mistakes and that their

key finding is entirely conditional on the use of parametrically smoothed ROC

curves when calculating accuracy, rather than the standard empirical ROC curves

that dominate the literature. Fixing these mistakes results in a reversal of their claim

that theory-based models of escalation are better at predicting onsets of civil war

than other kinds of models. Their model is outperformed by several of the ad hoc,

putatively non-theoretical models they devise and examine. More importantly, we

argue that rather than trying to contrast the roles of theory and “atheoretical”

machine learning in predictive modeling, it would be more productive to focus on

ways in which predictive modeling and machine learning could be used to strengthen

extant predictive work. Instead, we argue that predictive modeling and machine

learning are effective tools for theory testing.

1

Department of Political Science, Predictive Heuristics, Seattle, WA, USA

2

Independent Researcher, Washington, D.C., USA

3

Department of Political Science, Duke University, Durham, NC, USA

4

Department of Political Science, University of Washington, Seattle, WA, USA

Corresponding Author:

Michael D. Ward, Duke University, Perkins 326, Durham, NC 27707, USA.

Email: michael.don.ward@gmail.com; michael.d.ward@duke.edu

Journal of Conflict Resolution

2021, Vol. 65(7-8) 1405-1426

ªThe Author(s) 2020

Article reuse guidelines:

sagepub.com/journals-permissions

DOI: 10.1177/0022002720982358

journals.sagepub.com/home/jcr

Keywords

civil wars, replication, mo deling, internal armed co nflict, prediction, forec asting,

civil war

Blair and Sambanis (2020, hereafter B&S) argue that theory is essential for creating

models that have high accuracy in forecasting civil war onset. Indeed, they assert

that with such theory forecasting is more accurate than has previously been possible.

We re-examine the empirical basis for the claims made in support of it. We find that

these claims to be unsupported and evidence presented for them to be incorrect.

Their theory-based escalation model does not do better than the alternatives they

examine. It does worse. The reason for this is that they made several mistakes in their

research procedure. Further, the performance results they report are based on

smoothed performance curves, not the empirical, unsmoothed curves. This provides

misleading results. In addition, two of the structural alternatives to their basic esca-

lation model were incorrectly implemented. We also found that the scoring of their

forecasts for the first half of 2016 was incorrectly performed using civil war inci-

dence, not onset. In what follows, we show the impact of these mistakes on the

conclusions.

B&S claim (page 3) to argue that a model informed by procedural theories of

escalation and de-escalation can predict the onset of civil wars “remarkably

accurately.” Indeed, B&S argue that this theoretical model outperforms four other

“more mechanical” alternatives. They also claim that the integration of structure

with process is better than alternatives over short forecasting windows. Third, they

preregistered the list of thirty countries that have the highest risk of civil war onset.

They claim that such prospective predictions are rare in the literature when, in fact,

they have been routine for many years with several prominent projects.

1

B&S claim

to be unique in assessing these forecasts. A qualitative analysis of their predictions

allows them to conclude that their model is robust. We will return to their analysis

later, after correcting the procedural mistakes we found in their research process.

Before proceeding, we quote B&S (page 24):

Our theoretically driven model generates accurate forecasts, with base specification

AUCs of 0.82 and 0.85 over one- and six-month windows, respectively, and AUCs as

high as 0.92 in other specifications. Our model also consistently and sometimes dra-

matically outperforms the alternatives we test. [ ...] Cederman and Weidmann (2017,

476) argue that “the hope that big data will somehow yield valid forecasts through

theory-free ‘brute force’ is misplaced in the area of political violence.” Our results lend

some credence to this claim.

Although their analysis in the end does not support this point, we do not disagree

that theory can be important in predictive modeling. Of the various ways one could

contrast ad hoc machine learning with “theoretical” modeling, B&S focus

1406 Journal of Conflict Resolution 65(7-8)