Learn by doing: making your forecasts better.

• Author: Kavanagh, Shayne C.

This article is adapted from Informed Decision Making through Forecasting: A Practitioner's Guide to Government Revenue Analysis by Shayne Kavanagh and Daniel Williams (GFOA: 2017), available at www.gfoa.org/forecastbook.

To build expert skill, you need clear and unambiguous feedback on your performance, and a willingness to use that feedback to improve on specific areas of weakness. (1) Forecasting revenues is no different; research shows that the strongest predictor of improved forecasting performance is a deliberate, committed practice +of self-improvement, including updating the forecaster's personal assumptions and belief system about forecasting as a result of experienced (2)

The purpose of the financial forecast is to evaluate current and future fiscal conditions to guide policy and budgeting decisions. A financial forecast is a fiscal management tool that presents estimated information based on past, current, and projected financial conditions. This will help identify future revenue and expenditure trends that may have an immediate or long-term influence on government policies, strategic goals, or community services. The forecast is an integral part of the annual budget process. An effective forecast allows for improved decision making in maintaining fiscal discipline and delivering essential community services.

This article will show you how to analyze your forecasts well. You'll learn ways to use your experience with past forecasts to improve the two central aspects of forecast performance:

* Accuracy: The difference between forecasted and actual revenue.

* Effectiveness: The extent to which the forecast influences actual decisions.

KEEPING GOOD RECORDS

To perform a good analysis, you'll need to keep good records. Hold onto the following information:

* Data. You can get unambiguous feedback on how your forecasts have performed by comparing actual revenues to the original forecast. Hang onto your original forecasts, any updates, and long-term forecasts, and use them to evaluate how useful the original forecast was. Retain not just the forecast numbers but also the other data that will help you learn from experience, including the background material necessary to understanding how the forecast was calculated--especially an explanation of the quantitative method used.

* Assumptions. Also hold onto critical assumptions or judgments, or other quantities used in calculating the forecast, and check the accuracy of these underlying assumptions. If they were accurate, but the accuracy of the forecast was still unsatisfactory, the problem probably lies with the forecasting technique selected or how it was executed.

* Adjustments. If you made any manual adjustments, determine whether they're adding to or detracting from the accuracy of the forecast (research suggests it is often the latter). Also think about the adjustments you made and why you made them; if you don't understand the full story, you can't improve the forecast.

* Policy Changes. Policy changes are sometimes made after a forecast is issued--sometimes because of the information provided by the forecast--in an effort to affect the amount of revenue the government receives. For example, decision makers might change tax or fee rates, expand or contract the tax base, or modify the technical rules for a tax or fee computation. Keep track of these changes so you can adjust the assumptions in your model to judge how accurate the forecast would have been if the policy changes had been in effect when the forecast was issued.

FOCUSING ON REVENUE

Because most of your audience will be primarily concerned with the government's ability to annually budget expenditures in line with available revenues, focus your evaluation on the annual forecast for budgeting purposes.

The forecast should be evaluated in aggregate (e.g., all general fund revenues) as well as disaggregated (i.e., separate revenue sources). The aggregate forecast error is important because that is the forecast that ultimately affects budgeting decisions. In many cases, however, the aggregate forecast tends to be more accurate than any single individual forecast. That's because of the diversified revenues that comprise it, along with the consequent opportunity for a negative forecast error in one area to cancel out a positive forecast error in another. Examine forecast accuracy for individual revenue sources to get a complete picture of how the forecast is performing. For example, individual forecasts that all produce relatively small errors are much more reassuring than individual forecasts that produce relatively large but offsetting errors.

CALCULATING MAPE

You can develop accuracy statistics to help your tracking over time. The most useful accuracy statistic is mean absolute percentage error (MAPE). To calculate it:

1. Express the "error" (the difference between actual and...