Backtesting & data

What backtesting is for

Backtesting is checking a specific, rule-based idea against historical data to estimate its edge. The goal is not a perfect number — it is an honest answer to "does this setup, applied by my rules, have positive expectancy across a real sample, and how does it behave?" It replaces hope with evidence, and it exposes the ideas that only worked in your imagination.

The traps that fake an edge

Backtesting is riddled with ways to fool yourself. The professionals are defined by how carefully they avoid these:

• Hindsight / look-ahead bias. Using information you would not have had in real time (the day's close to judge a morning entry). If your rule peeks at the future, its results are fiction.
• Cherry-picking. Testing only the periods or instances that flatter the idea. Test every instance in a continuous window, losers included.
• Overfitting. Adding conditions until the strategy fits the past perfectly — a curve traced onto history that shatters on new data. A robust edge is simple and works across many conditions; an overfit one has ten rules that each "fixed" one bad trade.
• Ignoring costs. Spreads, slippage, and commissions turn many "profitable" backtests into losers. Include them.

Data and sample

An edge you can trust needs enough varied data: many instances, across different regimes (positive and negative gamma, calm and volatile). A strategy that only worked in one quiet quarter has not been tested — it has been observed once. And past performance never guarantees the future: markets change, edges decay. Backtesting builds a reasonable expectation, not a promise. Treat a good backtest as permission to trade small and keep measuring, not as a guarantee to size up.

Common mistakes

• Look-ahead bias. Any rule that uses future information invalidates the entire test.
• Overfitting. Ten conditions that perfect the past will fail the future; favor simple and robust.
• Skipping costs and ugly periods. A test without spreads, slippage, and the bad weeks is a fairy tale.

Forward testing — the honest complement

A backtest tells you how an idea would have done. Forward testing tells you how you do with it, and the gap between those two is where most strategies actually fail.

Running a setup live, in small size or on paper, exposes things history cannot. Can you recognise the setup in real time, without the benefit of the completed candle? Can you execute it when it appears at an inconvenient moment? Does the fill you assumed actually exist? Does the psychological experience of the drawdown match what the equity curve implied? A strategy that backtests beautifully and that you cannot execute is not an edge; it is a documented regret.

Forward testing also catches the subtlest form of self-deception: a rule that seemed perfectly objective in hindsight but requires a judgement call in the moment. If two reasonable people watching the same live chart would disagree about whether the setup fired, the rule is not yet specified.

Sample size, plainly

The uncomfortable truth about validation is that meaningful conclusions need more data than most traders want to collect.

A handful of trades tells you essentially nothing — a random process with no edge produces winning streaks routinely. Dozens of trades start to be suggestive. Hundreds, spread across different market conditions, begin to be evidence. And the noisier the strategy — the more variable the results — the more trades you need before the average means anything.

Two practical implications follow. First, be far more sceptical of a strategy with twenty examples than feels natural, however good the twenty look. Second, favour setups that occur often enough to accumulate a sample in reasonable time; a beautiful setup that appears four times a year cannot be validated within a career.

And when the sample is unavoidably small, size accordingly. Uncertainty about your edge is itself a risk, and the correct response to it is smaller positions rather than more confidence.

What to measure, beyond the equity curve

An equity curve is the least informative summary of a strategy, because it collapses everything into one line. Five other numbers tell you far more about whether an edge is real and whether it is yours.

Expectancy in R — average result per trade. The core number.

Win rate and average win/loss separately. The same expectancy can come from many wins and small edges or few wins and large ones, and those two are psychologically completely different to trade.

Maximum consecutive losses in the sample. This is your preview of the worst stretch, and it should drive your sizing.

Maximum drawdown in R. Combined with your risk per trade, this tells you the account impact to expect.

Distribution of results by context — regime, time of day, day of week. This is the most under-used and most valuable, because it frequently reveals that the edge lives entirely in one context and the other trades are dilution.

That last one alone has saved more strategies than any refinement. A setup that is strongly positive in a dampened regime and slightly negative otherwise is not a mediocre strategy — it is a good strategy with a missing filter.

The out-of-sample discipline

The single most effective technical safeguard against fooling yourself is to hold back data you have not looked at.

Develop and refine on one period. Then test, once, on a period you have never examined. If the results hold roughly, you have real evidence. If they collapse, you fitted the first period — which is the most common outcome, and finding it out on held-back data costs nothing compared with finding it out live.

The discipline is that the held-back test happens once. If you test, adjust and re-test on the same out-of-sample data, it stops being out-of-sample — you are simply fitting a second period more slowly. This rule is easy to state and genuinely hard to follow, because a failed out-of-sample test is disappointing and the temptation to "just adjust one thing" is enormous.

When the backtest and reality diverge

Eventually you will trade a validated strategy and get worse results than the test suggested. Four explanations cover nearly every case, and diagnosing which one applies is far better than concluding the market changed.

Costs. Spread, slippage and commissions were understated, or the fills assumed do not exist at your size.

Execution. You are not taking every signal — skipping the uncomfortable ones, entering late, exiting early. Your journal's rule-following column answers this definitively.

Regime. The live period is a different environment from the test period. Your context breakdown tells you whether this is plausible.

The edge was never there. Overfitting, small sample, or a subtle look-ahead. Painful, common, and better discovered at small size.

Note that three of the four are diagnosable from records you should already be keeping. Without those records, every divergence looks like the fourth explanation, and traders abandon working systems on the strength of a feeling.

Walk-forward testing, in plain terms

The most robust validation method has an intimidating name and a simple idea, and it is worth understanding even if you never implement it formally.

Instead of developing on all your data and testing on a held-back slice once, walk forward: develop on the first period, test on the next, then move the whole window forward and repeat. Develop on January to March, test on April. Then develop on February to April, test on May. And so on.

What this produces is a series of out-of-sample results across many different market environments, rather than one test in one environment. If the strategy holds up across most windows, you have far stronger evidence than a single successful out-of-sample period — which might simply have been a favourable regime.

It also reveals something a single test hides: whether the strategy's parameters keep changing. If the optimal settings differ wildly from window to window, the strategy is fitting noise rather than capturing something real, however good each individual window looks.

For a discretionary trader this is rarely done formally, but the principle transfers directly: evaluate your results in chunks across different periods rather than as one aggregate. A strategy that made all its money in one exceptional month is telling you something different from one that produced steady results across six.

The metrics that mislead

Three commonly quoted statistics deserve scepticism, because each can look excellent while describing something fragile.

Win rate. Easily inflated by taking small profits and letting losses run, which produces a beautiful percentage and a declining account. Meaningless without the average win-to-loss ratio beside it.

Total return. Says nothing about the path or the risk taken to get there. A return produced with occasional enormous positions is not comparable to the same return produced with consistent sizing, and only the second is repeatable.

The best-looking equity curve. Among many tested variations, the smoothest curve is frequently the most overfitted — it looks smooth precisely because it was selected for fitting the historical noise closely. The second- or third-best variant is often the more honest choice.

The metrics that mislead least are expectancy in R, maximum consecutive losses, and the distribution of results across different contexts. None of them are exciting, and all of them describe things that will actually happen to you.

• Backtesting estimates an edge from history — its value depends entirely on honesty.
• Kill look-ahead bias, cherry-picking, overfitting, and cost-blindness.
• Require a large, varied sample across different regimes.
• A good result licenses small live trading and measurement, not a size-up or a guarantee.