How Informative are the Subjective Density Forecasts of Macroeconomists?

Date01 April 2014
DOIhttp://doi.org/10.1002/for.2281
Published date01 April 2014
How Informative are the Subjective Density Forecasts of
Macroeconomists?
GEOFF KENNY,
1
*THOMAS KOSTKA
1
AND FEDERICO MASERA
2
1
European Central Bank, Frankfurt, Germany
2
Economics Department, Universidad Carlos III de Madrid, Getafe, Spain
ABSTRACT
In this paper, we propose a framework to evaluate the subjective density forecasts of macroeconomists using micro
data from the euro area Survey of Professional Forecasters (SPF). A key aspect of our analysis is the use of evaluation
measures which take account of the entire predictive densities, and not just the probability assigned to the outcome that
occurs. Overall, we nd considerable heterogeneity in the performance of the surveyed densities at the individual level.
However, it is hard to exploit this heterogeneity and improve aggregate performance by trimming poorly performing
forecasters in real time. Relative to a set of simple benchmarks, density performance is somewhat better for GDP
growth than for ination, although in the former case it diminishes substantially with the forecast horizon. In addition,
we report evidence of an improvement in the relative performance of expert densities during the recent period of
macroeconomic volatility. However, our analysis also reveals clear evidence of overcondence or neglected risks in
expert probability assessments, as reected in frequent occurrences of events which are assigned a zero probability.
Copyright © 2014 John Wiley & Sons, Ltd.
key words forecast evaluation; real-time data; Survey of Professional Forecasters
INTRODUCTION
Forecast users, including macroeconomic policymakers, require accurate and reliable information about the future
state of the economy. However, the recent nancial crisis and its associated heightened levels of macroeconomic
volatility have underlined the limitation of point forecasts as a sufcient information variable. Indeed, as surveyed
in Tay and Wallis (2000), a large forecasting literature exists on the importance of density forecasts and the need
to supplement forecast information on expected future values of forecast target variables with additional information
from the forecast variables entire predictive density. For example, central banks and scal policymakers often rely as
muchif not moreon assessments of the risks to the macroeconomic outlook that can be extracted from density
forecasts. Also, in the newly prioritized area of macro prudential supervision and nancial stability analysis, a key
aspect underpinning the assessments of vulnerabilities in the nancial sector and their systemic implications includes
a review and identication of relevant macroeconomic risks that are extracted from the entire predictive densities for
the variables of interest.
In all of the above contexts, it is crucial that forecast users and decision makers have access to evidence on the
overall quality and accuracy of such information. For example, are macroeconomic experts able to provide risk
and uncertainty predictions that are superior to simple rules of thumb or even very naïve statistical statements, e.g.
equivalent to ippinga coin? Additionally, are macroeconomic experts better ableto assess the uncertainty surrounding
some economic variables, such as output growth, rather than others such as ination? Or do such risk assessments have
equivalent validityor information content, at both short- and longer-term horizons or during particular business cycle
episodes? Answers to such questions should be of interest to policymakers or anyone else who relies on expert risk
assessments when making their decisions and economic choices.
In this paper, as a way of shedding some light on these questions, we propose and implement various methods to
evaluate the accuracy of the information contained in the surveyed density forecasts of macroeconomists.
1
The
surveyed forecasts that we examine refer to macroeconomic developments in the euro area and are collected as part
of the ECB Survey of Professional Forecasters (SPF).
2
The study of such real-time forecasts, as collected in
surveys, came to prominence with the work of Zarnowitz (1969) and Zarnowitz and Lambros (1987), and contrasts
with much of the applied forecasting literature that focuses on the pseudoreal-time predictions of speci c models
which have often not been available for actual decision making. In an earlier study, Diebold et al. (1999) employ the
*Correspondence to: Geoff Kenny, DG Research, European Central Bank, Kaiserstrasse 29, D-60311 Frankfurt, Germany. E-mail: Geoff.
kenny@ecb.europa.eu
1
The focus of the paper is therefore very much on evaluating the accuracy of such surveyed forecasts, in particular with reference to the observed
outcomes. This contrasts with another strand in the literature studying the impact of beliefs, expectations and uncertainty more generally on the
macroeconomy. Recent examples of this strand of literature include Rich et al. (2012) or Bloom (2009).
2
Additional information about the survey as well as the complete underlying micro dataset can be downloaded from http://www.ecb.europa.eu/
stats/prices/indic/forecast/html/index.en.html. See Bowles et al. (2010) for detailed background information related to the ECB SPF.
Journal of Forecasting,J. Forecast. 33, 163185 (2014)
Published online 3 March 2014 in Wiley Online Library (wileyonlinelibrary.com) DOI: 10.1002/for.2281
Copyright © 2014 John Wiley & Sons, Ltd.
probability integral transform to evaluate the densities from the US SPF. Another strand in the literature on density
evaluation, as surveyed in Tay and Wallis (2000), and represented by Wallis (2003), Dowd (2008) and Knüppel and
Schulterfrankenfeld (2012), has focused on evaluation of the density forecasts produced and published by central
banks. The latter study has, for example, attempted to test empirically the optimality of measures of skew extracted
from central bank density forecasts, nding that they have little systematic information content.
3
An important feature of our study is the use of individual-level data from the euro area SPF in gauging the
accuracy of macroeconomic density forecasts. To conduct our evaluation, we assess the predictive performance of
the surveyed densities relative to various crude benchmark alternatives. The chosen benchmarks are based on simple
distributional assumptions and condition only on information that was publicly available at the time the survey was
carried out. They therefore represent a somewhat low-lying threshold against which to assess the skillor incremental
information content, embodied in the subjective density forecasts of macroeconomic experts. The logic behind this
approach implies that to the extent that professional economists can deliver insightful information on macroeconomic
risks, e.g. regarding skew or heightened probabilities of more extreme events, they should be able to outperform these
simple and mechanical benchmarks. Our focus on individual density forecasts and not just on the distribution of point
forecasts or even the combined density forecast that is often reported in ofcial publications helps avoid a possible bias
that can result from examining aggregate forecasts alonea concern which is even more relevant when one is
aggregating entire distributions and not merely point forecasts.
4
To take account of this potential for aggregation bias,
when seeking to draw inference about the ability of macroeconomists to provide informative assessments, we report
separately the evaluation results for the cross-section of individual survey replies together with the results for the
aggregation of individual probabilities.
The layout of the remainder of the paper is as follows. In the next section we describe in detail the evalu ation
framework we employ. The third section provides some summary information on distributional properties of the sur-
veyed density forecasts we examine. The fourth section presents the main empirical results evaluating and
interpreting the performance of the density forecasts for GDP growth and HICP ination at both 1- and 2-year-ahead
horizons. Finally, the fth section concludes with our overall assessment of the ability of macroeconomists to forecast
risks as reected in the relative performance of their surveyed density forecasts and makes some suggestions for
future research aimed at enhancing further the usefulness of such surveyed forecasts.
EVALUATING DENSITY FORECASTS
The traditional approach to testing the information content of density forecasts in macroeconomics has been to use the
probability integral transform (PIT), as surveyed in Diebold et al. (1998) and following earlier work by Dawid (1984)
and Rosenblatt (1952). However, the latter is particularly problematic in small samples, as is usually the cas e in
macroeconomic applications, given that the null hypothesis of a well-behaved density only yields large-sample
predictions for the statistical properties of the PIT. In addition, for multi-period forecasts, under the null that the
predictive density coincides with the true but unobserved density, the distributional features of the PITs are not well
dened. As a result, it is only possible to test the null of a well-behaved density forecast jointly with an assumed
model of the process driving the associated multi-period PITs (see, for example, Dowd, 2008). As an alternative to
the PIT framework for assessing the information content of a given density forecast, we propose examining tests
of equal predictive ability between the subjective expert forecasts and a predened benchmark forecast using the
framework of Diebold and Mariano (1995) and as extended in Giacomini and White (2006). In particular, we
examine a null hypothesis (H
0
) of equal predictive ability between the surveyed densities and a benchmark density
forecast conditional on the information set available to forecasters at the time the forecast was made.
5
To the extent
that the surveyed density forecasts have lower overall loss relative to the chosen benchmark, rejection of H
0
would
represent statistical evidence supporting the marginal information value in the density forecasts and risk perceptions
of macroeconomists. In the remainder of this section, we discuss our design of this test with a view to its subsequent
application to the surveyed densities from the ECB SPF.
3
Boero et al. (2011) and Casillas-Olvera and Bessler (2006) provide a comparative evaluation of central bank and private sector density forecasts
collected in surveys.
4
Indeed, we would argue that a meaningful answer to the question posed in the title of the paper can only be based on the individual level fore-
casts. This reects the fact that, as discussed in Wallis (2005), Geweke and Aminsano (2011) and Hall and Mitchell (2007), the act of aggregating
densities into a linear opinion pool transforms the individual densities along most dimensions that are relevant to the question at hand. In partic-
ular, as a nite mixture distribution, the equal weighted combination of density forecasts will tend to have a variance that is larger than the average
variance of its constituent densities, reecting the impact of disagreement across models or survey participants on the location of the true under-
lying density. The aggregate density may also exhibit other properties such as positive or negative skew and/or fat tailseven if the underlying
individual densities do not exhibit such features. Engelberg et al. (2011) have recently emphasized the potential for bias in the interpretation of
aggregated surveyed forecasts, linked to temporary instability in the respondents to the survey. Rich and Tracy (2010) consider alternative mea-
sures of uncertainty using micro data from the US SPF.
5
Giacomini and White (2006) argue that this notion of conditional predictive ability more closely represents the real-time problem confronting
forecasters in practice.
164 G. Kenny et al.
Copyright © 2014 John Wiley & Sons, Ltd. J. Forecast. 33, 163185 (2014)

To continue reading

Request your trial

VLEX uses login cookies to provide you with a better browsing experience. If you click on 'Accept' or continue browsing this site we consider that you accept our cookie policy. ACCEPT