Robust forecast aggregation: Fourier L2E regression

• Author: David W. Scott,Barbara Mellers,Philip E. Tetlock,Daniel Cross,Jaime Ramos
• Published date: 01 April 2018
• Date: 01 April 2018
• DOI: http://doi.org/10.1002/for.2489

Received: 20 August 2016 Revised: 3 June 2017 Accepted: 17 August 2017

DOI: 10.1002/for.2489

RESEARCH ARTICLE

Robust forecast aggregation: Fourier L2Eregression

Daniel Cross1Jaime Ramos2Barbara Mellers3Philip E. Tetlock3David W. Scott1

1Department of Statistics, Rice University,

Houston, TX, USA

2Department of Statistics, Indiana

University, Bloomington, IN,USA

3Department of Psychology,University of

Pennsylvania, Philadelphia, PA,USA

Correspondence

Daniel Cross, Department of Statistics,

Rice University, 6100 Main St., Houston,

TX 77005, United States.

Email: dc23@rice.edu

Abstract

The Good Judgment Team led by psychologists P. Tetlock and B. Mellers of

the University of Pennsylvania was the most successful of five research projects

sponsored through 2015 by the Intelligence Advanced Research Projects Activ-

ity to develop improved group forecast aggregation algorithms. Each team had

at least 10 algorithms under continuous development and evaluation over the

4-year project. The mean Brier score was used to rankthe alg orithms on approx-

imately 130 questions concerning categorical geopolitical events each year. An

algorithm would return aggregate probabilities for each question based on the

probabilities provided per question by thousands of individuals, who had been

recruited by the Good Judgment Team. This paper summarizes the theorized

basis and implementation of one of the two most accurate algorithms at the con-

clusion of the Good Judgment Project. The algorithm incorporated a number

of pre- and postprocessing steps, and relied upon a minimum distance robust

regression method called L2E. The algorithm was just edged out by a variation of

logistic regression, which has been described elsewhere. Work since the official

conclusion of the project has led to an even smaller gap.

KEYWORDS

brier score, constrained data blurring, minimum distance criterion, probability extremization and

normalization, variance-stabilizing transformation

1INTRODUCTION

The problem of political prediction is a serious endeavor

for students of history. The modern media is replete with

experts and pundits who offer their judgment of the

likelihood of future events. Tetlock (2005) summarizes a

long-term study of the accuracy of such forecasts, and

found surprisingly poor performance by such experts. The

Intelligence Advanced Research Projects Activity (IARPA)

initiated a competition among teams organized by uni-

versities to evaluate the accuracy of aggregate forecasts

over thousands of individuals for hundreds of short-

and medium-term scenarios, called individual forecasting

problems (IFP). The Good Judgment Team (GJT) was by

the far the most successful in terms of accuracy of pre-

dictions, based upon the smaller Brier score. The inner

workings of the GJT are described Ungar et al. (2012). A

subset of the forecasters was identified as consistently pro-

viding low Brier scores. These so-called “super forecasters”

truly performed in a consistently high manner.Tetlock and

Gardner (2015) provide a summary of the overall project.

Among the dozens of algorithmic approaches for ana-

lyzing the reams of data from the thousands of individuals

who provided forecasts for every question, two algorithms

were consistently in the top two spots at the forecast aggre-

gation task. The individuals' forecasts were not just yes/no,

but actual probability vectors for the two to five options

available in each IFP scenario. Individuals were free to

update their forecasts at any time, as new information was

acquired. A variation of logistic regression eventually took

top honors. Close behind was a robust regression approach

that began with a minimum distance criterion (integrated

Journal of Forecasting. 2018;37:259–268. wileyonlinelibrary.com/journal/for Copyright © 2017 John Wiley & Sons, Ltd. 259