What Backtesting Is, and Why Naive Backtests Lie

A backtest is the first thing most people build when they get interested in algorithmic trading, and it’s the first thing that fools them. The idea is simple and reasonable: take a trading rule, run it over historical price data, and see what would have happened. The problem is that “what would have happened” is much harder to reconstruct honestly than it looks, and almost every shortcut you take bends the result in your favor.

Before anything else, the framing that keeps you sane: a backtest is not a prediction and not a guarantee. It’s a hypothesis test against one particular slice of the past. At best it tells you a rule wasn’t obviously broken on data it has seen. It cannot tell you the rule will work on data it hasn’t. Everything below is about the gap between those two statements.

A quick note: this is educational material, not trading advice. Algorithmic crypto trading is high-risk and most retail algo traders lose money.

Lookahead bias: letting the strategy peek

Lookahead bias is using information in a decision that you would not have had at the moment of the decision. It is the single most common reason a backtest looks brilliant and then dies in production.

The classic version is off-by-one indexing on the bar. Suppose you decide to buy when today’s candle closes above its 20-day moving average. If your code computes the signal from the closing price and then fills the order at that same close, you’ve cheated: in reality you only know the close once the bar is over, so the earliest you could trade is the next bar’s open. That one-bar shift sounds trivial. On fast, mean-reverting crypto data it can be the entire edge.

Subtler versions hide everywhere. Normalizing your whole dataset by its mean and standard deviation before splitting leaks the future into the past. Filling missing values with a method that looks forward. Using a data point that was later revised, or that arrived with a delay you didn’t model. The discipline is to ask of every value your strategy touches: would I actually have known this number, in this form, at this timestamp?

Survivorship bias: the coins that vanished

Survivorship bias is testing only on the assets that still exist. Crypto is an extreme case. Thousands of tokens have gone to zero, been delisted, or quietly stopped trading. If your dataset is “the top coins as they exist today,” you’re testing exclusively on winners and near-winners — the survivors — and silently excluding everything that blew up.

A momentum strategy backtested only on survivors looks fantastic, because by construction every asset in the sample went on to have a future. Real trading includes the ones that didn’t. To avoid this you need point-in-time universe data: the set of tradeable assets as it was on each historical date, including the ones that later died. That data is harder to get and less glamorous, and skipping it is why so many crypto backtests are science fiction.

Overfitting: torturing the data until it confesses

Give yourself enough parameters and enough attempts and you can fit any curve. If you try 500 combinations of lookback windows, thresholds, and filters, some of them will look spectacular on your history purely by chance — the same way that, out of enough coin-flippers, someone flips ten heads in a row. Overfitting is mistaking that luck for skill.

Two habits fight it:

Hold out data you never touch. Split your history into a training period, where you develop and tune, and a test period, which you look at exactly once at the end. If the strategy only works in-sample, it was noise. For time series, prefer walk-forward validation: tune on a window, test on the next window, roll forward, and repeat, so you’re always testing on data that came after the tuning.
Prefer fewer knobs and a reason. A strategy with two parameters and an economic rationale for why it should work is far more trustworthy than one with nine parameters that happen to line up. Complexity is where overfitting hides.

The costs that quietly eat the edge

Even a bias-free signal can be unprofitable once reality’s frictions are included, and naive backtests love to ignore them.

Fees. Every trade pays exchange fees. A high-frequency strategy that trades hundreds of times can hand its entire theoretical profit to the exchange.
Slippage. The price you model is rarely the price you get. Market orders walk the order book; your fill is worse than the last printed price, especially in size or in thin markets.
Spread. Buying at the ask and selling at the bid costs you the spread on every round trip. On illiquid tokens the spread alone can dwarf the edge.
Market impact. Big enough orders move the price against you. A backtest assuming you can trade any size at the quoted price is imagining a market that doesn’t exist.

A useful gut check: re-run the backtest with deliberately pessimistic costs. If the edge survives conservative fees and slippage, it’s more likely real. If a strategy is only profitable at zero cost, it isn’t profitable.

What an honest backtest looks like

Put the defenses together and a trustworthy backtest has a recognizable shape. It decides on one bar and fills on the next, never on data from the future. It runs on a point-in-time universe that includes assets that later died. It’s developed on a training set and confirmed once on untouched test data, ideally walk-forward. It has few parameters, each with a rationale. And it charges itself realistic fees, spread, and slippage.

Do all of that and the returns you see will be far less exciting than the naive version — which is exactly the point. A modest, robust result you believe is worth more than a spectacular one you can’t trust. The goal of a backtest was never to produce a big number. It was to give you an honest reason to think a rule might survive contact with a live market — and, just as often, to talk you out of one that won’t.