Reassessing False Discoveries in Mutual Fund Performance: Skill, Luck, or Lack of Power?

• Date: 01 October 2019
• Author: ANGIE ANDRIKOGIANNOPOULOU,FILIPPOS PAPAKONSTANTINOU
• DOI: http://doi.org/10.1111/jofi.12784
• Published date: 01 October 2019

THE JOURNAL OF FINANCE •VOL. LXXIV, NO. 5 •OCTOBER 2019

Reassessing False Discoveries in Mutual Fund

Performance: Skill, Luck, or Lack of Power?

ANGIE ANDRIKOGIANNOPOULOU and FILIPPOS PAPAKONSTANTINOU∗

ABSTRACT

Barras, Scaillet, and Wermers propose the false discovery rate (FDR) to separate skill

(alpha) from luck in fund performance. Using simulations with parameters informed

by the data, we find that this methodology is conservative and underestimates the

proportion of nonzero-alpha funds. For example, 65% of funds with economically large

alphas of ±2% are misclassified as zero alpha. This bias arises from the low signal-

to-noise ratio in fund returns and the resulting low statistical power. Our results

question FDR’s applicability in performance evaluation and other domains with low

power, and can materially change the conclusion that most funds have zero alpha.

IN AN INFLUENTIAL STUDY, Barras, Scaillet, and Wermers (2010)—hereafter

BSW—propose the false discovery rate (FDR) as a methodology for separating

skill from luck and precisely estimating the proportions of funds that generate

true “alpha.” Applying this approach to U.S. equity mutual funds, they find

that the vast majority (75%) of funds have zero-alpha net of expenses, a sizable

minority (24.4%) have negative alpha, and only a negligible proportion (0.6%)

beat the benchmarks. These findings have been widely cited in the literature

as evidence of no skill in the industry, and have been interpreted as consistent

with the Berk and Green (2004) equilibrium.1But the contribution of BSW

stretches beyond the mutual fund literature and extends to introducing and

popularizing the FDR methodology in finance. The remarkable accuracy of the

FDR estimator as shown by BSW in a simulation—together with the approach’s

simplicity—has spurred a number of studies to apply it further in the context

of fund performance as well as in other contexts. For example, it has been used

∗Angie Andrikogiannopoulou and Filippos Papakonstantinou are at King’s Business School,

King’s College London. This manuscript has grown out of simulations and ideas that were pre-

viously part of a paper circulated under the title “A Direct and Full-Information Estimation of

the Distribution of Skill in the Mutual Fund Industry.” The authors thank Enrico Biffis; Markus

Brunnermeier; Pasquale Della Corte; Michael Dempster; Robert Kosowski; Alex Kostakis; Alex

Michaelides; Elias Papaioannou; Fabio Trojani; Dimitri Vayanos;seminar participants at BI Nor-

wegian Business School, Citadel LLC, Imperial College London, Lancaster University,Manchester

Business School, and the University of Geneva; and conference participants at the 2012 Cambridge-

Princeton conference, the 2014 European Finance Association meeting, and the 2014 European

Seminar on Bayesian Econometrics. The authors have no conflicts of interest as identified by the

Journal of Finance’s disclosure policy.

1See,forexample,Busse,Goyal,andWahal(2010), Ben-Rephael, Kandel, and Wohl(2012), and

Jiang, Verbeek, and Wang (2014).

DOI: 10.1111/jofi.12784

2667

2668 The Journal of Finance R

to assess the performance of trading strategies, to estimate the proportion of

takeovers that experience abnormal trading volume, and to detect jumps in

asset returns.2

In this study, we reassess whether the FDR methodology can successfully

distinguish skill from luck in mutual funds. Expanding the simulation of BSW,

we find that for data-generating processes (DGPs) that are informed by mutual

fund data, the FDR estimator becomes markedly biased. This bias arises from

the fact that the pivotal assumptions behind the estimator fail due to the low

signal-to-noise ratio in fund return data and the resulting lack of statistical

power in tests of fund alpha. In particular, our simulations show that, given

the information in the data, the FDR methodology misclassifies as zero-alpha

many funds with economically large alphas (e.g., ±2% per year) and hence

may greatly underestimate the proportion of nonzero-alpha funds. We also

find that while the number of observations per fund affects the estimator’s

accuracy, the number of funds itself does not, as it does not affect the signal-to-

noise ratio. This distinction is important, as most applications of the FDR in

finance involve panels with a large Nbut small T. Thus, while the simulation

in BSW is a valuable first step in assessing the FDR methodology for fund

performance evaluation, it does not diagnose these limitations because it is

conducted under the specific assumptions that all nonzero alphas are very large

(around 3.5% per year) and that there are a large number of observations per

fund.

The bias that we demonstrate calls into question the economic conclusions

of the FDR approach for fund alpha. Specifically, the finding that most mutual

funds have (almost) zero alpha may not be due to a lack of skill in the indus-

try and hence may not support the Berk and Green (2004) model, in which

decreasing returns to scale and rational capital reallocation drive fund alphas

to zero, but rather is likely an artifact of an estimation methodology that has

low power to detect nonzero-alpha funds.3Overall, our results raise concerns

about the applicability of the FDR in fund performance evaluation and in other

areas in finance in which the signal-to-noise ratio in the data is similarly

low.

The FDR: The FDR approach was developed by Benjamini and Hochberg

(1995) in statistics to control the proportion of null hypotheses that are falsely

rejected when conducting multiple tests. As a less conservative alternative to

previous approaches such as the Bonferroni correction, use of the FDR has

2Cuthbertson, Nitzsche, and O’Sullivan (2012) and Criton and Scaillet (2014) apply the FDR

in the context of U.K. mutual funds and hedge funds, respectively; Bajgrowicz and Scaillet (2012)

apply it in the context of trading strategies; Augustin, Brenner, and Subrahmanyam (2019) apply

it in the context of takeovers; Patton and Ramadorai (2013) use it to assess funds’ risk exposures;

and Bajgrowicz, Scaillet, and Treccani (2015) use it to detect jumps in asset returns.

3To clearly see that FDR analysis of real mutual fund data yields biased estimates, one need

only compare the proportion of funds it classifies as skilled/unskilled on the basis of returns before

and after expenses (which average 1% per year): it estimates that 75% of funds have zero alpha

after expenses, which would imply that at least as many have positive alpha before expenses, but

it estimates that only 10% of funds have positive alpha before expenses.