Robust model rankings of forecasting performance

• Author: Prasad Sankar Bhattacharya,Dimitrios D. Thomakos
• Date: 01 September 2018
• Published date: 01 September 2018
• DOI: http://doi.org/10.1002/for.2529

RESEARCH ARTICLE

Robust model rankings of forecasting performance

Prasad Sankar Bhattacharya

1

| Dimitrios D. Thomakos

2

1

Department of Economics, Deakin

University, 70 Elgar Road, Burwood,

Victoria 3125, Australia

2

Department of Economics, University of

Peloponnese, Tripolis 22 100, Greece

Correspondence

Dimitrios D. Thomakos, Department of

Economics, University of Peloponnese,

Tripolis 22 100, Greece.

Email: thomakos@uop.gr

Abstract

This paper investigates robust model rankings in out‐of‐sample, short‐horizon

forecasting. We provide strong evidence that rolling window averaging

consistently produces robust model rankings while improving the forecasting

performance of both individual models and model averaging. The rolling

window averaging outperforms the (ex post) “optimal”window forecasts in

more than 50% of the times across all rolling windows.

KEYWORDS

exchange rate forecasting, inflation forecasting, model averaging, model ranking, robustness, rolling

window

1|INTRODUCTION

This paper investigates potential ranking of forecasting

methods with the key aim of establishing robust evidence

in the ranking procedure. The main idea is that window

averaging may improve model rankings with superior

forecasting performance for individual models and model

averaging. Model averaging, that is, combining forecasts

from different models, is proposed earlier to improve

predictability.

1

However, potential benefits of averaging

across models and sample periods (rolling windows) for

the same model are not being explored fully. The choice

of window, namely recursive or rolling, in improving

forecasts is also widely debated. A number of papers cite

efficiency to select recursive windows (Inoue & Kilian,

2006; Meese & Rogoff, 1983; Stock & Watson, 1999). In

contrast, Inoue, Jin, and Rossi (2014) employ rolling

windows to improve predictability since rolling estima-

tion may address the problem of parameter instability.

Recent methodological advances focus on either

selecting optimal rolling windows to generate better

forecasting performance (Inoue et al., 2014; Pesaran &

Timmermann, 2007) or producing tests of forecasting

performance robust to the choice of that window (Hansen

& Timmermann, 2012; Rossi & Inoue, 2012). Clark and

McCracken (2009) use recursive and rolling windows

from a single model to improve forecasts. The above

papers partially address issues related to different seg-

ments of information, producing results which may affect

decision making in the presence of a higher degree of

uncertainty. In addition, various fragments of informa-

tion may lead to model rankings that are not robust over

time (Engel & West, 2005). Giacomini and Rossi (2009)

propose a forecast breakdown test for potential structural

breaks in parameters after addressing possible instability

in the distribution of the regressors and predicting future

forecast breakdowns.

However, if the aim is to produce optimal forecasts

with robust model rankings, then forecast breakdown

tests or a robust choice of a single sample split may not

be helpful. It is difficult to know a priori the model

performances with different time segments as the data

generating process is changing over time. Rather, it is

preferable to consider various time segments and average

1

Clemen (1989) reviews a number of combining applications in econom-

ics and other fields. Stock and Watson (2004) and Timmermann (2006)

provide overviews on forecast improvements using model averaging.

Aiolfi, Capistran, and Timmermann (2011) argue that model forecast

combinations outperform individual model forecasts in the presence of

unstable parameters. Clements and Hendry (2006) and Pesaran and

Timmermann (2005) identify model instability adversely affecting fore-

casting performance and propose model forecast combinations in

improving forecasts.

Received: 2 January 2012 Revised: 19 October 2017 Accepted: 12 April 2018

DOI: 10.1002/for.2529

676 Copyright © 2018 John Wiley & Sons, Ltd. Journal of Forecasting. 2018;37:676–690.wileyonlinelibrary.com/journal/for