Why misleading backtests feel so convincing
Software makes it easy to iterate: change a parameter, rerun history, watch the curve improve. Without discipline, you are no longer testing a hypothesis. You are fitting a story to the past.
The goal of a backtest is not the prettiest chart. It is a realistic estimate of whether an idea deserves demo or small live risk. Keep that goal visible when the optimiser tempts you.
1. Overfitting
Tuning many parameters until the past looks perfect usually captures noise rather than an edge. The strategy becomes a custom-fit costume for one historical path, then fails when the market changes clothes.
Prefer fewer parameters, hold out out-of-sample data, and check that performance is stable across nearby parameter values. If a tiny tweak destroys results, you probably fitted noise.
- Limit free parameters before you start optimising
- Reserve a final unseen period you do not touch until the end
- Prefer plateaus of good settings over a single lucky peak
2. Look-ahead bias
Using information that would not have been available at decision time inflates results. Classic examples: entering at the open using that same bar’s close, or reading an indicator that secretly includes future data.
Make sure signals only use completed bars and that your software’s bar-replay or Strategy Tester mode matches how you will trade live. If the platform can “see” the whole candle before you decide, your edge is fictional.
3. Survivorship bias
Testing only on companies that exist today ignores those that were delisted or went bankrupt. Your historical basket looks healthier than the real opportunity set traders faced at the time.
For stock strategies, use datasets that include delisted securities when possible. For other markets, still ask whether your symbol list is a cleaned winner’s list or a realistic universe.
4. Ignoring trading costs
Commissions, spreads, slippage and borrowing costs can turn a profitable high-frequency idea into a losing one. Many default backtests understate friction, especially around news, thin liquidity and stop entries.
Model costs conservatively. If the strategy only works with perfect fills and zero spread, it does not work. Re-run with harsher cost assumptions and see whether the edge survives.
5. Too few trades
A handful of trades says little about a strategy. Ten beautiful winners can be luck. You want a sample large enough to be statistically meaningful and spread across different market regimes: trends, ranges, high and low volatility.
If your timeframe cannot produce enough trades, widen the history window, test related symbols, or accept that you do not have enough evidence yet.
6. Unrealistic fills
Assuming every limit order fills, or that you can trade illiquid instruments at the mid price, overstates performance. Stop orders that always fill at the stop price with no slippage are another common fantasy.
Check how your software models fills in the mode you use (bar open, OHLC, every tick, real ticks). Visual replay helps discretionary traders spot whether their assumed entry was actually available.
7. Not testing robustness
Before you trust a result, stress it. A strategy that only works on one year, one symbol, or one exact parameter set is fragile. Robustness checks will not guarantee live profits, but they catch many soft illusions early.
- Walk-forward analysis or a clear in-sample / out-of-sample split
- Monte Carlo resampling of trades or equity paths
- Testing on different markets and time periods
- Varying parameters, costs and starting dates
A practical checklist before you go live
Ask four questions: Did I use information I would not have had live? Did I include realistic costs? Do I have enough trades across regimes? Did results hold when I changed settings and periods?
If any answer is no, keep researching. Follow promising backtests with forward testing on demo or a small live account before you scale risk.