Pricing Model Performance and the Two‐Pass Cross‐Sectional Regression Methodology

• DOI: http://doi.org/10.1111/jofi.12035
• Author: CESARE ROBOTTI,JAY SHANKEN,RAYMOND KAN
• Published date: 01 December 2013
• Date: 01 December 2013

THE JOURNAL OF FINANCE •VOL. LXVIII, NO. 6 •DECEMBER 2013

Pricing Model Performance and the Two-Pass

Cross-Sectional Regression Methodology

RAYMOND KAN, CESARE ROBOTTI, and JAY SHANKEN∗

ABSTRACT

Over the years, many asset pricing studies have employed the sample cross-sectional

regression (CSR) R2as a measure of model performance. We derive the asymptotic dis-

tribution of this statistic and develop associated model comparison tests, taking into

account the impact of model misspecification on the variability of the CSR estimates.

We encounter several examples of large R2differences that are not statistically sig-

nificant. A version of the intertemporal capital asset pricing model (CAPM) exhibits

the best overall performance, followed by the Fama–French three-factor model. Inter-

estingly,the performance of prominent consumption CAPMs is sensitive to variations

in experimental design.

THE TRADITIONAL EMPIRICAL METHODOLOGY for exploring asset pricing models en-

tails estimation of asset betas (systematic risk measures) from time-series fac-

tor model regressions, followed by estimation of risk premia via cross-sectional

regressions (CSRs) of asset returns on the estimated betas. In the classic anal-

ysis of the capital asset pricing model (CAPM) by Fama and MacBeth (1973),

a CSR is run each month, with inference ultimately based on the time-series

mean and standard error of the monthly risk premium estimates. Also see the

related paper by Black, Jensen, and Scholes (1972).

∗Kan is from the University of Toronto. Robotti is from the Federal Reserve Bank of Atlanta

and EDHEC Risk Institute. Shanken is from Emory University and the National Bureau of Eco-

nomic Research. We thank Pierluigi Balduzzi; Christopher Baum; Jonathan Berk; TarunChordia;

Wayne Ferson; Nikolay Gospodinov; Olesya Grishchenko; Campbell Harvey (the Editor); Ravi Ja-

gannathan; Ralitsa Petkova; Monika Piazzesi; Yaxuan Qi; Tim Simin; Jun Tu; Chu Zhang; Guofu

Zhou; two anonymous referees; an anonymous Associate Editor; an anonymous advisor; semi-

nar participants at the Board of Governors of the Federal Reserve System, Concordia University,

Federal Reserve Bank of Atlanta, Federal Reserve Bank of New York, Georgia State Univer-

sity, Penn State University, University of New South Wales, University of Sydney, University of

Technology, Sydney, and University of Toronto; as well as participants at the 2012 Utah Winter

Finance Conference, the 2009 Meetings of the Association of Private Enterprise Education, the

2009 CIREQ-CIRANO Financial Econometrics Conference, the 2009 FIRS Conference, the 2009

SoFiE Conference, the 2009 Western Finance Association Meetings, the 2009 China International

Conference in Finance, and the 2009 Northern Finance Association Meetings for helpful discus-

sions and comments. Kan gratefully acknowledges financial support from the Social Sciences and

Humanities Research Council of Canada and the National Bank Financial of Canada. The views

expressed here are the authors’ and not necessarily those of the Federal Reserve Bank of Atlanta

or the Federal Reserve System.

DOI: 10.1111/jofi.12035

2617

2618 The Journal of Finance R

A formal econometric analysis of the two-pass methodology was first pro-

vided by Shanken (1992). He shows how the asymptotic standard error of the

second-pass risk premium estimator is influenced by estimation error in the

first-pass betas, requiring an adjustment to the traditional Fama–MacBeth

standard errors. A test of the validity of the pricing model’s constraint on ex-

pected returns can also be derived from the CSR residuals (see, for example,

Shanken (1985)).

As a practical matter, however, models are at best approximations of reality.

It is therefore desirable to have a measure of “goodness of fit” with which to as-

sess the performance of a risk-return model. The most popular measure, given

its simple intuitive appeal, has been the R2for the cross-sectional relation.

This R2indicates the extent to which the model’s risk measures (betas) ac-

count for the cross-sectional variation in average returns, typically for a set of

asset portfolios. The R2for average returns is employed in this context, rather

than the average of monthly R2s, since the latter could be high, with positive

ex post risk premia for some months and negative premia for others, even if

the ex ante (average) premium is zero.

A recent paper by Lewellen, Nagel, and Shanken (2010) explores the sam-

pling distribution of the R2estimator via simulations. However, despite its

widespread use in conjunction with the two-pass methodology, the cross-

sectional R2has been treated mainly as a descriptive statistic in asset pricing

research. We take an important step beyond this limited approach by deriving

the asymptotic distribution of the R2estimator.

Ultimately, though, researchers are interested in comparing models, and so

it is also important to determine the distribution of the difference between R2s

for competing models. This issue appears to have been completely neglected

in the literature thus far, even in simulations. Again, we provide the rele-

vant asymptotic distribution and find, through a series of simulations, that it

provides a good approximation of the actual sampling distribution. The simu-

lation analysis employs 50 years of monthly data, consistent with much em-

pirical practice. Our main econometric analysis of model comparison based

on R2parallels that in Kan and Robotti (2009), who focus exclusively on the

Hansen and Jagannathan (1997, HJ hereafter) distance, an alternative mea-

sure of model fit. In addition, we explore, for the first time, model comparison

based on R2in an excess returns specification with the zero-beta rate con-

strained to equal the risk-free rate. Finally, we derive asymptotic tests of mul-

tiple model comparison, that is, we evaluate the joint hypothesis that a given

model dominates a set of alternative models in terms of the cross-sectional

R2.

All of our procedures account for the fact that each model’s parameters must

be estimated and that these estimates will typically be correlated across mod-

els. Both ordinary least squares (OLS) and generalized least squares (GLS) R2s

are considered. OLS is more relevant if the focus is on the expected returns for

a particular set of assets or test portfolios, but the GLS R2may be of greater in-

terest from an investment perspective. In this regard, Kandel and Stambaugh

(1995) show that there is a direct relation between the GLS R2and the relative

Pricing Model Performance and the Two-Pass CSR Methodology 2619

efficiency of a market index. They also argue, as do Roll and Ross (1994), that

there is virtually no relation at all for the OLS R2unless the index is exactly

efficient.1

Model comparison essentially presumes that deviations from the implied

restrictions are likely for some or all models. This “misspecification” might

be due, for example, to the omission of some relevant risk factor, imperfect

measurement of the factors, or failure to incorporate some relevant aspect of

the economic environment—taxes, transaction costs, irrational investors, etc.

Thus, misspecification of some sort seems inevitable given the inherent limi-

tations of asset pricing theory. Yet researchers often conduct inferences about

risk premia or other asset pricing model parameters imposing the null hypoth-

esis that the model is correctly specified. Indeed, it is not uncommon to see this

done even when a model is clearly rejected by the data—a logical inconsistency.

Therefore, the asymptotic properties of the two-pass methodology are derived

here under quite general assumptions that allow for model misspecification,

extending the results of Hou and Kimmel (2006) and Shanken and Zhou (2007)

under normality.

Empirically, our interest is in rigorously evaluating and comparing the per-

formance of several prominent asset pricing models based on their cross-

sectional R2s. In addition to the basic CAPM and consumption CAPM

(CCAPM), the theory-based models considered are the CAPM with labor

income of Jagannathan and Wang (1996), the CCAPM conditioned on the

consumption–wealth ratio of Lettau and Ludvigson (2001), the ultimate con-

sumption risk model of Parker and Julliard (2005), the durable consumption

model of Yogo (2006), and the five-factor implementation of the intertempo-

ral CAPM (ICAPM) used by Petkova (2006). We also study the well-known

three-factor model of Fama and French (1993). Although this model was

primarily motivated by empirical observation, its size and book-to-market

factors are sometimes viewed as proxies for more fundamental economic

factors.

Our main empirical analysis uses the “usual” 25 size and book-to-market

portfolios of Fama and French (1993) plus five industry portfolios as the as-

sets. The industry portfolios are included to provide a greater challenge to

the various asset pricing models, as recommended by Lewellen, Nagel, and

Shanken (2010). We limit ourselves to five industry portfolios since some of

our asymptotic results become less reliable as the number of test portfolios in-

creases. Specification tests reject the hypothesis of a perfect fit for the majority

of the models, so that statistical methods that are robust to model misspeci-

fication are clearly needed. We show empirically that misspecification-robust

standard errors can be substantially higher than the usual ones when a fac-

tor is “nontraded,” that is, is not some benchmark portfolio return. As one

example, consider the t-statistic on the GLS risk premium estimator for the

consumption growth factor in the durable consumption model of Yogo (2006).

1See Lewellen, Nagel, and Shanken (2010) for a related multifactor GLS result with “mimicking

portfolios” substituted for factors that are not returns.