What a backtest gets wrong about fills

A backtest is a claim that a set of decisions, applied to history, would have produced a particular result. Most of the attention goes to the decisions. Much less goes to the assumption sitting underneath them: that when the system wanted to trade, it traded, at the price it was looking at.

That assumption is almost never neutral. The ways it fails are systematic rather than random, and they overwhelmingly fail in the direction that makes the backtest look better than the system.

The order you sent is not the order that competed

A limit order resting at a price is not entitled to a fill when the market touches that price. It is entitled to its position in a queue. Whether it fills depends on how much size sits ahead of it, how much of that size cancels rather than trades, and how much volume arrives before the price moves away.

A backtest that fills a limit order because the printed low reached the limit price is asserting a queue position it never modelled. On a liquid instrument at a busy price level, an order placed late in the book's life may see thousands of contracts trade at its price and never fill at all.

The error is not symmetric, which is the important part. When the price merely touches your level and reverses, you are unlikely to have filled. When the price cuts decisively through your level, you certainly did. So the fills a naive backtest grants you most generously are exactly the ones where the market kept going against you afterwards. Fills are selected on the cases where they hurt.

That is adverse selection, and it is not a small correction. A passive strategy that looks profitable under a touch-fills-you assumption can be structurally unprofitable once queue position is modelled, because the entire measured edge was made of fills that would not have happened.

OHLC bars describe a path that never existed

Bar data gives four prices and no ordering between the high and the low. A strategy evaluated on bars can be made to look extraordinary by resolving that ambiguity in its favour (assuming entries at the low and exits at the high of the same bar), and the resolution is often implicit rather than deliberate.

The subtler version is a stop and a target both sitting inside one bar's range. Which one triggered is not recoverable from the bar. Whatever the engine assumes, it assumes for every such bar in the sample, and a rule that resolves the tie optimistically will accumulate an edge that exists only in the ambiguity.

Neither problem is fixable by being careful with bars. It requires the path (trades and quotes in sequence) or a conservative rule that resolves every ambiguity against the strategy and accepts the resulting understatement.

Signal time is not order time

Between the bar closing and the order reaching the matching engine there is data handling, computation, network transit and the venue's own queue. A backtest that executes at the close of the bar that generated the signal has assumed all of that is zero.

How much this matters is entirely a function of the holding period. On a multi-week position it is noise. On something reacting to an event, it can be most of the move: the price at the moment the signal was computable and the price at the moment an order could arrive are different prices, and the difference is worst exactly when the signal was strongest, because that is when everyone else's orders arrived too.

Size is a price, not a quantity

The top of book is a quantity, and it is frequently smaller than the position a backtest assumes it acquired instantly at that price. Filling beyond it means walking the book, and the average fill is worse than the quote that was displayed.

Two distinct costs follow. The mechanical one is that a larger order fills at a worse average price, which is arithmetic and easy to model badly but at least easy to see. The second is that trading moves the price, and a strategy whose backtest assumes otherwise has assumed its own trades are invisible. For a strategy that concentrates its trading at a particular time of day or around a particular event, that assumption fails hardest precisely where the strategy is most active.

Partial fills are a state, not an edge case

A backtest usually treats an order as filled or not filled. Live, an order is frequently filled in part, and the remainder is a position that has to be managed: left working, cancelled and re-sent, or crossed. Each choice has a cost and each produces a different realised price.

The engineering consequence is larger than the pricing one. A system that has no representation of a partially filled order has no correct behaviour when it encounters one, and it will encounter one. This is the class of failure that does not show up in research at all and shows up on the first difficult day in production.

What to do about it

None of this argues for abandoning backtests. It argues for treating the execution assumption as a modelled quantity with its own error bar, rather than as plumbing.

• Resolve every ambiguity against the strategy. Fill at the worse side, assume the adverse ordering within the bar, take the later timestamp. A strategy that survives systematic pessimism is worth more than one tuned against a favourable assumption.
• Model queue position, or do not trade passively in the backtest. If the data to estimate queue position is not available, a passive strategy has not been tested.
• Sweep the fill assumption as a parameter. Vary spread capture, latency and participation, and look at the shape of the response, not one number. If the result collapses under a plausible assumption, the sensitivity is the finding.
• Reconcile against live fills as soon as any exist. The difference between modelled and realised fill price, tracked per order, is the only direct measurement of how wrong the assumption was. It is also the fastest way to discover the system is not doing what the research said.

The general shape here is the same one that appears elsewhere in this firm's method. A backtest reports a level. What matters is how that level responds when the assumptions underneath it are moved, and an execution model is one of the assumptions most capable of moving it.