Price Measurement Using Scanner Data: Time‐Product Dummy Versus Time Dummy Hedonic Indexes

• Published date: 01 June 2021
• Author: Jan Haan,Rens Hendriks,Michael Scholz
• Date: 01 June 2021
• DOI: http://doi.org/10.1111/roiw.12468

© 2020 The Authors. Review of Income and Wealth published by John Wiley & Sons Ltd on behalf of

International Association for Research in Income and Wealth

394

PRICE MEASUREMENT USING SCANNER DATA: TIME-PRODUCT

DUMMY VERSUS TIME DUMMY HEDONIC INDEXES

by Jan de Haan

Division of Corporate Services, IT and Methodology, Statistics Netherlands, and OTB, Faculty of

Architecture and the Built Environment,Delft University of Technology

Rens HendRiks

Statistics for Development Division, Pacic Community (SPC)

AND

MicHael scHolz*

University of Graz

This paper compares two model-based multilateral price indexes: the time-product dummy (TPD) index

and the time dummy hedonic (TDH) index, both estimated by expenditure-share weighted least squares

regression. The TPD model can be viewed as the saturated version of the underlying TDH model, and

we argue that the regression residuals are “distorted toward zero” due to overfitting. We decompose

the ratio of the two indexes in terms of average regression residuals of the new and disappearing items.

The decomposition aims to explain the conditions under which the TPD index suffers from quality-

change bias or, more generally, lack-of-matching bias. An example using scanner data on packaged

men's T-shirts illustrates our framework.

JEL Codes: C43, E31

Keywords: hedonic regression, multilateral price indexes, new and disappearing items, quality change,

scanner data

1. intRoduction

The advent of scanner data, and other electronic “big data” such as web-

scraped data, has increased the potential for accurate price measurement well

beyond the traditional method of price collectors visiting outlets and collecting

prices for a relatively small sample of products. Electronic data usually comprise

all the products sold by a certain retailer, and in the case of scanner data, quantities

sold, therefore product weights, are available. Detailed characteristics are some-

times readily available as well, and if they are not, they can often be extracted from

websites.

Note: The authors would like to thank Johan Verburg for research assistance and Michael Webster

for valuable comments. The views expressed in this paper are those of the authors and do not necessar-

ily reflect the views of Statistics Netherlands or SPC.

*Correspondance to: Michael Scholz, Department of Economics, University of Graz,

Universitätsstraße 15/F4, 8010 Graz, Austria (michael.scholz@uni-graz.at).

Review of Income and Wealth

Series 67, Number 2, June 2021

DOI: 10.1111/roiw.12468

This is an open access article under the terms of the Creative Commons Attribution License, which

permits use, distribution and reproduction in any medium, provided the original work is properly

cited.

bs_bs_banner

Review of Income and Wealth, Series 67, Number 2, June 2021

395

© 2020 The Authors. Review of Income and Wealth published by John Wiley & Sons Ltd on behalf of

International Association for Research in Income and Wealth

The matched-models method for constructing price indexes is inappropriate

for scanner data, especially in areas where there is a regular churn in models

or where prices are changed only when new models are introduced in the mar-

ket. For such product markets, the Consumer Price Index (CPI) Manual (ILO/

IMF/OECD/UNECE/Eurostat/The World Bank, 2004) recommended the use

of explicit quality adjustments using hedonic regression techniques that take

advantage of the characteristics data. The CPI Manual, however, did not dis-

cuss the issue of drift that can occur in period-on-period chained weighted price

indexes and the more recently proposed multilateral index number methods to

remove the chain drift while maximizing the number of matches in the data.

There is a range of multilateral methods; see Diewert and Fox (2017) and Chessa

etal. (2017).

de Haan (2015) proposed the use of two related model-based multilateral

price indexes for incorporating scanner data, both estimated by expenditure-share

weighted least squares regression: the time dummy hedonic (TDH) index where

information on item characteristics is available, and the time-product dummy

(TPD) index when this information is lacking. The name TPD method was sug-

gested by de Haan and Krsinich (2014) as it adapts Summers’ (1973) multilateral

country-product dummy (CPD) method for spatial comparisons to price compar-

isons across time. A potential problem with the TPD method is that the resulting

price index will not be explicitly adjusted for quality change. The aim of the present

paper is to examine what drives the difference between the two methods.

We build on work by Silver and Heravi (2005) and Krsinich (2016). Silver and

Heravi (2005) compared TPD and TDH indexes but only in a period-on-period

chained context, where the bilateral TPD index equals a matched-model index.

Krsinich (2016) argued that the use of longitudinal price information makes the

multilateral TPD index implicitly quality-adjusted; see also Aizcorbe etal. (2003).

It is true that in many cases (though perhaps not in oligopolistic markets with

strategic pricing aimed at particular market segments) price differences among

coexisting items provide us with information about the value of quality differ-

ences. Nevertheless, as with any implicit quality-adjustment method, this does not

necessarily imply proper treatment of new and disappearing items and therefore

does not rule out the possibility of quality-change bias or, more generally, lack-of-

matching bias.

The treatment of quality change is especially important where there is a sub-

stantial churn in items sold when new models are introduced and old ones dis-

appear. A major reason for the difference between the TPD and TDH methods

potentially arises from the likely shortfall in the implicit quality adjustments of

the TPD method compared to the more robust explicit quality adjustments of the

TDH method, which are based on quality characteristics. Also, the TPD method

cannot deal with items that are new in the sample period (these items lie exactly

on the regression surface and are “zeroed out”); the TDH method does account

for these items. The magnitude of the difference between the two methods will

depend on the degree of churn, the extent of the quality difference between the

new and disappearing models, and the adequacy of the characteristics data used

in the hedonic regression to capture price-determining quality differences. Against