How to Spot Backtest Overfitting: A 20-Question Checklist

Backtest overfitting is tuning a trading strategy until it fits the random noise in past data instead of a real effect. You can spot it by asking how many variations were tried, whether the result survives on markets and periods it was not designed on, whether costs and realistic fills are included, and whether its returns fell after the idea was published or launched. A backtest that cannot answer those questions is a hypothesis, not evidence.

The scale of the problem is documented. In 2016 Campbell Harvey, Yan Liu and Heqing Zhu catalogued 316 factors proposed in leading finance journals to predict stock returns and concluded that the usual hurdle, a t-statistic of about 2, was far too low; a new factor should clear about 3.0. The same year David McLean and Jeffrey Pontiff found that 97 published predictors earned about 58% less after publication than in the original papers. That is peer-reviewed research. A vendor's backtest faces no review at all.

This guide is educational. It explains how to read evidence and recommends no strategy.

Why does the best backtest always look good?

Because testing many ideas manufactures winners. If a strategy has no true edge, its measured Sharpe ratio over T years is roughly normally distributed around zero, with a standard deviation of about 1 divided by the square root of T. Test enough strategies and the best one's Sharpe rises steadily with the number of tries, even though every one of them is worthless.

The companion's worked example simulates 1,000 strategies with no edge at all over five years.

Stylized simulation; computed in the companion.

The best backtest cleared a demanding statistical hurdle and was still pure luck. When there is no edge, last period's winner is last period's luck.

What happens when one real edge hides among many variants?

This is the realistic case: a vendor starts from a sound idea and then tunes it. The book's chapter on measuring edge simulates one strategy with a genuine but modest edge, a true Sharpe ratio of 0.5, tested alongside variants with none, using eight years of data. The researcher picks the best backtest and trades it for three years.

Stylized; 4,000 repetitions per row; returns assumed normal, so fat tails would make every row worse.

Four points follow.

An honest test of a real edge looks unimpressive. Tested once over eight years, a true Sharpe of 0.5 has a median t-statistic of 1.41, failing even the old standard of 2.

The search manufactures significance. Test 1,000 variants and the winner clears a t-statistic of 3.21, yet it is the genuine strategy only 4% of the time.

The backtest overstates the live result. With 50 variants, the median live Sharpe is 0.10 against a backtest of 0.82, a haircut of about 87%.

Three years of live trading decide very little. Even the genuine strategy loses money over three years 20% of the time, so a short live record cannot easily tell a real edge from a fitted one.

The practical rule: the number of trials sets the hurdle. Researchers forget the variants they discarded and vendors do not say, so assume the number is large.

What are the most common backtest biases?

Overfitting is one bias among several. The companion's study of backtesting, overfitting and the evidence problem lists the main ones.

Costs alone can reverse a result. A stylized example in the companion takes a strategy earning 8% a year gross at 12% volatility. At one turn a year with 10 basis points a round trip, it keeps 7.9%. At 100 turns a year, it loses 2.0% a year.

Options backtests are especially fragile. Historical options prices are usually end-of-day quotes, and the midpoint of a wide quote on an illiquid strike was never a tradable price. Instruments must also have existed: SPX options have expired every trading day only since May 2022, so a backtest of same-day SPX options before then is fiction.

How to check a backtest: 20 questions

The companion's checklist groups twenty questions into five parts. Each has a good answer and a bad one.

Data

1. Does the data include dead securities and funds? Good: a named, survivorship-free source. Bad: "current S&P 500 members back to 2000."
2. Is it point-in-time? Good: index membership and fundamentals as known on each date. Bad: today's constituents and restated accounts.
3. Did the instruments exist on each date? Good: start dates noted. Bad: products tested before they launched.
4. For options, are prices traded bids and asks on strikes with open interest? Bad: model prices or mid-quotes.

Rules

5. What is the source of edge, and who is on the other side? Good: a named premium, behavior, flow, information or speed advantage. Bad: "the indicator shows it."
6. Were the rules fixed and written down before the test?
7. How many parameters, and how sensitive is the result? A result that collapses when a lookback changes from 14 to 15 is a fitted optimum.
8. Could the signal have been known at the time of the trade? Signal and trade at the same close is the most common look-ahead bug.

Statistics

9. How many strategies and variations were tried? "We don't track that" is the bad answer.
10. Is there a multiple-testing adjustment such as the deflated Sharpe ratio?
11. How long is the sample? Ten or more years and a t-statistic well above 3, given the trials.
12. Does it work in markets and periods it was not designed on?
13. What is the worst drawdown, how long did it last, and when?
14. What is the skew, and does the sample include a crisis? Short-volatility strategies look best right before they fail.

Costs and execution

15. Which costs are included, at the trader's size? "Commissions included" alone is not enough.
16. How are fills modeled? Market orders should pay the spread; limit orders should fill only if the price trades through.
17. What is the capacity?
18. Are margin calls and forced liquidation modeled?

Live evidence

19. Is there a live record, and who verifies it? Audited statements or an exchange benchmark index, not screenshots.
20. Has performance decayed since publication or launch, and what would make you stop?

The companion's scoring is blunt. Eighteen to twenty good answers is strong evidence for a backtest, still worth a small live trial. Eight to twelve is weak, a hypothesis rather than a system.

How should you read a live track record?

A live record beats a backtest because real money was traded, but it still needs reading. Four traps are common.

Gross or net? Renaissance Technologies' Medallion fund is reported to have earned about 66% a year gross and about 39% net, unaudited. Fees can more than halve a gross return.

What is the denominator? Universa's reported 3,612% in March 2020 was the return on the capital its tail hedge required. A portfolio with 3.33% in Universa and the rest in the S&P 500 gained about 0.4% that month, by Institutional Investor's calculation.

How long, and through which crises? A five-year record needs a Sharpe ratio of about 1.34 to clear a t-statistic of 3. A short-option record without a crash says little about the crash.

Is it too smooth? Infinity Q's smooth returns came from variance swaps its founder mismarked; he was sentenced to 180 months in April 2023. A record too smooth for its strategy is a warning.

Red flags in trading system marketing

Any of these moves a claim to the lowest evidence grade until shown otherwise: returns quoted per week or month with no drawdown or dates; a win rate without the average loss; "hypothetical" in small print or not at all; screenshots instead of statements; "AI" or "institutional secret" as the stated edge; a product that costs more than the edge would earn on a typical account; and a seller who earns from participants' failures, as with retail prop-firm challenges.

Even real edges decay

Passing every test does not make an edge permanent. The companion split US factor returns from the Kenneth French data library at the publication of each defining paper, gross of costs.

Every factor earned less after publication. That is why the catalogue moved momentum from A to B and graded size alone F. A sensible planning rule is roughly half a paper's published return before costs, and less for anomalies in small, costly stocks. AQR's value drawdown of 2018–2020 shows the other side: a real edge can look broken for years. The chapter on finding an edge you can name covers how to tell the difference.

Backtests in the four factors of the edge equation

The Edge Equation argues that trading profit needs four factors that multiply: edge, survival, execution, and discipline and adaptation. Overfitting is a failure of the first. A system with no true edge multiplies everything else by zero, however well it is sized, executed and followed. It also undermines the fourth: a trader who cannot tell a decaying edge from a normal drawdown will either quit a sound system at its worst or keep trading a dead one. The catalogue's grades for all 118 systems are in the chapter on the systems scorecard. For how evidence fits into the wider business, see our guide to how trading firms make money, and for sizing a strategy that passes, our guide on how to calculate risk of ruin.

The takeaway

A backtest is the cheapest experiment in finance, and the best of many backtests is almost always flattered by luck. The questions that expose overfitting are simple: how many things were tried, does it work where it was not designed, are costs and fills realistic, and what happened after publication or launch. Plan on a fraction of any published return, and decide before going live what result would show the edge has gone. Chapter 4 of The Edge Equation sets out the evidence ladder and the statistics in full.