Presidential Address: The Scientific Outlook in Financial Economics

• Published date: 01 August 2017
• Date: 01 August 2017
• Author: CAMPBELL R. HARVEY
• DOI: http://doi.org/10.1111/jofi.12530

The Journal of Finance R

Campbell R. Harvey

The President of the American Finance Association 2016

THE JOURNAL OF FINANCE •VOL. LXXII, NO. 4 •AUGUST 2017

Presidential Address: The Scientific Outlook

in Financial Economics

CAMPBELL R. HARVEY∗

ABSTRACT

Given the competition for top journal space, there is an incentive to produce “sig-

nificant” results. With the combination of unreported tests, lack of adjustment for

multiple tests, and direct and indirect p-hacking, many of the results being published

will fail to hold up in the future. In addition, there are basic issues with the interpre-

tation of statistical significance. Increasing thresholds may be necessary,but still may

not be sufficient: if the effect being studied is rare, even t>3 will produce a large num-

ber of false positives. Here I explore the meaning and limitations of a p-value. I offer

a simple alternative (the minimum Bayes factor). I present guidelines for a robust,

transparent research culture in financial economics. Finally,I offer some thoughts on

the importance of risk-taking (from the perspective of authors and editors) to advance

our field.

SUMMARY

rEmpirical research in financial economics relies too much on p-values, which

are poorly understood in the first place.

rJournals want to publish papers with positive results and this incentivizes

researchers to engage in data mining and “p-hacking.”

rThe outcome will likely be an embarrassing number of false positives—effects

that will not be repeated in the future.

rThe minimum Bayes factor (which is a function of the p-value) combined with

prior odds provides a simple solution that can be reported alongside the usual

p-value.

rThe Bayesianized p-value answers the question: What is the probability that

the null is true?

rThe same technique can be used to answer: What threshold of t-statistic do I

need so that there is only a 5% chance that the null is true?

rThe threshold depends on the economic plausibility of the hypothesis.

LET ME START WITH AN ORIGINAL EMPIRICAL EXAMPLE. Some of you are familiar

with my work on identifying factors. Consider a new factor that is based on

equity returns from CRSP data. The t-statistic of the long-short portfolio is 3.23,

∗Campbell R. Harvey is with Duke University and the National Bureau of Economic Research. I

appreciate the comments of Manuel Adelino, Geert Bekaert, Jim Berger, Alon Brav, Anna Cieslak,

WayneFerson, David Hirshleifer, Jay Im, Ben McCartney,Stefan Nagel, Brenda Priebe, Emmanuel

Roman, Stephan Siegel, Michael Stutzer, Joy Tong, and Ivo Welch. I am especially grateful to

Yan Liu for comments on many versions of this paper. Jay Im and Jane Day provided research

assistance.

DOI: 10.1111/jofi.12530

1399

1400 The Journal of Finance R

which exceeds the Harvey, Liu, and Zhu (HLZ; 2016) recommended threshold

of 3.00. The factor has a near-zero beta with the market and other well-known

factors, yet an average return that comfortably exceeds the average market

excess return. What is this factor?

Here are the instructions that I gave my research assistant: (1) form portfo-

lios based on the first, second, and third letters of the ticker symbol; (2) show

results for 1926 to present and 1963 to present; (3) use a monthly, not daily,

frequency; (4) rebalance portfolios monthly and once a year; (5) value weight

and equally weight portfolios; (6) make a choice on delisting returns; and (7)

find me the best long-short portfolio based on the maximum t-statistic.

There are 3,160 possible long-short portfolios based on the first three let-

ters of the tickers. With two sample periods, there are 6,320 possible portfolio

choices, equal and value weights bring this number to 12,640, and two choices

for reconstituting the portfolio doubles this number again. In short, there are

a huge number of choices.

Many would argue that we should increase the choice space further because

there are other possible choices that I did not give to my research assistant.

Suppose, for instance, there are three ways to handle delisting returns. Ex

ante, one was chosen. The argument is that we should consider the fact that,

hypothetically, we could have had three choices, not just the one chosen (see

Gelman and Loken (2013)).

It is not surprising that, under a large enough choice set, the long-short strat-

egy has a “significant” t-statistic—indeed, dozens of strategies have “signifi-

cant” t-statistics. This is an egregious example of what is known as p-hacking.

One might think this is a silly example. But it is not. A paper referenced in

the HLZ (2016) factor list shows that a group of companies with meaningful

ticker symbols, like Southwest’s LUV, outperform (Head, Smith, and Watson

(2009)). Another study, this time in psychology, argues that tickers that are

easy to pronounce, like BAL as opposed to BDL, outperform in IPOs (Alter

and Oppenheimer (2006)). Yet another study, in marketing, suggests that tick-

ers that are congruent with the company’s name outperform (Srinivasan and

Umashankar (2014)). Indeed, some have quipped that ETF providers such as

Vanguard might introduce a new family of ETFs called “AlphaBet” with each

ETF investing in stocks with the same first letter of a ticker symbol.

Many interpret HLZ (2016) as suggesting that we “raise the threshold for dis-

covering a new finding to t>3.” However, the point of that paper is that many

in our profession (including myself) have been making an error in not adjusting

thresholds for multiple tests. In this address, I emphasize that making a deci-

sion based on t >3 is not sufficient either. In particular, raising the threshold for

significance may have the unintended consequence of increasing the amount of

data mining and, in turn, publication bias. Journals contribute to data mining

through their focus on publishing papers with the most “significant” results.

The reason is that journal editors are often competing for citation-based impact

numbers. Indeed, if you go to the American Finance Association’s homepage,

you will see the Journal of Finance’s impact factor prominently displayed. Be-

cause papers that do not report “significant” results generate fewer citations