Measuring Edge: Backtests, Track Records and the Evidence Problem

By the mid-2010s finance professors had proposed so many variables that seemed to predict stock returns that the collection had a name: the factor zoo. In "…and the Cross-Section of Expected Returns", published in the Review of Financial Studies in 2016 (volume 29, issue 1), Campbell Harvey, Yan Liu and Heqing Zhu catalogued the factors proposed in leading journals, 316 of them, and asked a simple question. If researchers had tried that many ideas on largely the same US data, how many of the successes were luck? Their answer was that the usual standard of evidence, a t-statistic of about 2, was far too low; a newly proposed factor should clear a t-statistic of about 3.0, and many published factors were probably false discoveries.

In the same year, in the Journal of Finance (volume 71, issue 1), David McLean and Jeffrey Pontiff published the other half of the problem. They took 97 variables that published studies had shown to predict returns and compared each variable's returns in the original paper's sample, in the period between the end of that sample and publication, and after publication. Returns were about 26% lower out of sample and about 58% lower after publication. They attributed the first fall to statistical bias in the original research and the further fall, about 32 points, to investors learning from the papers and trading on them. Four years later Kewei Hou, Chen Xue and Lu Zhang re-tested 452 anomalies with the smallest stocks controlled for and returns value-weighted; 65% failed a conventional significance test of |t| ≥ 1.96, and 82% failed at 2.78 (Review of Financial Studies 33(5), 2020).

These are results for peer-reviewed research, the second-best kind of evidence a trader can have. Vendor backtests, course sellers' screenshots and "AI bot" marketing face no review at all. If a professor's published anomaly loses more than half its return once it is known, the backtest on a trading-system website deserves much more doubt.