Evaluating the Energy-Saving Effects of a Utility Demand-Side Management Program: A Difference-in-Difference Coarsened Exact Matching Approach.

• Author: Boampong, Richard

1. INTRODUCTION

   Since the late 1970's, there has been a wide variety of Utility Demand-Side Management (DSM) programs to reduce energy consumption. Price-based programs such as peak-load pricing and incentive-based Demand Response (DR) programs such as direct load control, demand bidding, and interruptible programs are considered most effective in reducing peak period energy demand. However, most utilities find it difficult to implement these measures due to program cost and problems with overpayment or underpayment of incentives due to unverifiable baseline mechanisms for obtaining consumption reductions (Bushnell et al., 2009). Residential home retrofitting programs thus appear as an alternative for energy savings that can avoid the problems of price-based or incentives-based demand response programs. Also, these traditional energy efficiency retrofit programs can help install the automation systems needed to allow consumers to participate in an automated demand-response programs. Another significant advantage of residential retrofitting programs is that unlike price or incentive-based demand response programs, they "do not involve major adjustment to consumers' lifestyles and offer potential economic returns to consumers" (Ryan and Gamtessa, 2007). Currently, almost all electric utilities in the United States offer rebate programs to encourage customers to participate in retrofit programs.

   As these energy efficiency retrofit programs grow in size and cost, (1) there is the need to understand better their effects and cost-effectiveness. Since the 1990's, there has been a multitude of evaluation methodologies ranging from the crystal ball measures of savings (e.g., monthly energy savings in California's 20-20 program in the summer of 2005 was calculated as the difference in energy consumption relative to the same month in the previous year. (2)) to engineering simulation models, (3) and to various econometric models combining monthly meter readings and available data on customer characteristics to determine energy savings (e.g. Jones et al., 2010; Cohen et al., 1991). Engineering methods use simulations to predict energy savings from specific measures at the individual building level or the end-use equipment level. Since these engineering methods do not require customers' consumption data, they are theoretically appealing when customer information is not available. However, predictions from engineering models are normally flawed and misrepresent the actual energy savings since they do not account for the influences of confounding factors such as behavior and demographics of a household (Fels and Keating, 1993). Econometric methods, on the other hand, use consumers billing information while controlling for weather and household-level and building-level factors that might affect consumers' energy consumption.

   Most econometric evaluations of the effects of a DSM program use the classic difference-in-difference (DD) methodology or a variant of it where the impact of the DSM program is estimated as the difference in mean outcomes between all households participating in the program and those not participating (e.g. Godberg, 1986). This approach leads to bias if there are unobserved characteristics that affect the probability of participating in the program that are also correlated with the outcome of interest. Further, the result might also be biased if program participants are very different from non-participants in terms of their pre-treatment characteristics. Even controlling for pre-treatment characteristics in the DD regression does not necessarily reduce this bias since the estimated effect depends on the exact functional form used.

   In this paper, we evaluate the energy-saving effects of a residential retrofitting program, GRU's high-efficiency AC rebate program. We combine a difference-in-difference (DD) methodology with a Coarsened Exact Matching (CEM) approach (4) described in Iacus et al. (2012) to overcome the bias from confounding pre-treatment characteristics. Such an estimation approach is novel to the evaluation of energy savings from demand-side management programs, and they are appropriate for dealing with selection bias (5) when there are no valid instruments to allow for an instrumental variable approach. This method is also important to evaluating DSM programs for other reasons; matching on neighborhoods allows us to compare participants and non-participants in the same neighborhood. Hence, we are able to disentangle the effects of weather from program effects since houses in the same neighborhood are more likely to experience the same weather. Also, since houses built in the same year or a few years apart and in the same neighborhood are likely to be built with the same building materials and have similar characteristics, using neighborhoods and age of building in our matching methodology controls for the effects of building characteristics and materials on electricity consumption. An added importance of the CEM method is that since the rebate program had a very low participation rate, it provides a way to select a reasonable control group from the high percentage of non-participating households. For example, only about 6% of households participated in at least one of GRU's rebate programs in the year 2009. This percentage is much lower (about 2%) when we consider only the high-efficiency AC program. Using all the 98% of households that did not participate as a control group may bias the energy-saving estimate because the treatment group does not include all sections of the population.

   We use data on household electricity and natural gas consumption and retrofit program participation from Gainesville Regional Utilities from 2008 to 2012. Specifically, we evaluate the energy-saving effects of the 2009 high-efficiency AC rebate program. First, we estimate the energy-saving effect on annual energy consumption. Next, since the main aim of a DSM or energy efficiency programs is to reduce peak period consumption, we disaggregate the annual effect into summer peak effect, winter peak effect, and non-peak months effect to study the savings impact of the program on peak period energy consumption, particularly summer peak consumption. The results indicate that while the program led to substantial energy consumption reductions in the summer peak, winter peak reductions are small or non-existent. Also, by following the group of households that participated in the program for another year, we found no statistically significant rebound effects. This implies that the supply resources that the DSM program is designed to displace will indeed be avoided over the long run. The remainder of this chapter is as follows: Section 2 gives a brief background of GRU's rebate programs, Section 3 describes the empirical strategy, Section 4 gives a brief description of the data, and Section 5 investigates selection into treatment based on pre-treatment characteristics. Section 6 presents the results of the program on annual energy consumption and summer peak consumption while Section 7 gives an estimate of the rebound effect. Section 8 concludes.

2. BACKGROUND: GAINESVILLE REGIONAL UTILITIES REBATE PROGRAMS

   Gainesville Regional Utilities (GRU) offers its consumers a mix of rebates and incentives to promote energy efficiency. GRU offers rebates for high-efficiency central air conditioners, room air conditioning units, heat pumps, water heaters, insulation, duct sealing, refrigerator recycling, pool pumps, installation of solar water heaters, and attic measures. GRU also offers incentives for a comprehensive whole system measure through its Energy Star Home Performance Program and Low-Income Energy Efficiency Program. In this paper, we evaluate the energy-saving effect of the high-efficiency central air conditioner rebate program. The high-efficiency central air conditioner program encourages homeowners to replace their old, low-efficiency Heating Ventilation and Air-Conditioning (HVAC) system with a new high-efficiency unit. To qualify for the rebate, households must use a partnering Florida state licensed HVAC mechanical contractor in all retrofitting work. In 2009 about 3,226 single-family households (representing about 6% of all single-family homes in Gainesville) voluntarily participated in at least one of the rebate programs offered by GRU. Participants were allowed and even encouraged to participate in multiple rebate programs to maximize the energy savings. Table 1 lists the relevant financial incentives in GRU's 2009 rebate programs. (6)

3. EMPIRICAL STRATEGY AND METHOD

   This section motivates and summarizes our method. The aim is to overcome problems in the estimation of energy savings in the previous literature and also to provide a simple method of controlling for the effects of weather on electricity consumption when there is no proxy for household-specific weather. We use a difference-in-difference (DD) strategy in combination with the Coarsened Exact Matching (CEM) methodology described in Iacus et al. (2012).

   Let [treat.sub.it] [member of] {0,1} be an indicator of whether household i participated in the rebate program under consideration in period t and let [y.sub.it] be the electricity consumption of household i in period t. Let [y.sup.1.sub.it+s] be the electricity consumption of household i, s periods after participating in the rebate program. Also let [y.sup.0.sub.it+s] be the counterfactual electricity consumption of household i in period t + s had it not participated in the rebate program. Thus the gain or energy savings from participating in the rebate program for household i is:

   [[DELTA].sub.i] = [y.sup.1.sub.it+s] - [y.sup.0.sub.it+s]. (1)

   If we could simultaneously observe [y.sup.1.sub.it+s] and [y.sup.0.sub.it+s] for the same household, then program evaluation would be straightforward. We could estimate [[DELTA].sub.i] for every household that participated in the rebate program and...