How wrong are backtest queue models?

What I did

CME market-by-order (MBO) data tag every add, cancel, modify and fill with an order ID, so the first-in, first-out (FIFO) queue can be rebuilt exactly. Market-by-price (MBP) data show only total size per price, so when it drops without a trade a backtester must guess whether the cancel was ahead of it or behind.

I scored six rules, from risk_averse (hftbacktest's RiskAdverse: every cancel behind you) through hftbacktest's power and log models (including Rigtorp's rule) to front_optimistic (every cancel ahead). The truth is 45.5 million crude and 12.2 million gas real resting orders that joined the best price, over 141 crude and 125 gas sessions, March to October 2026.

The main metric is fill-time error: seconds between estimated and true fill, capped at 60 seconds, so false and missed fills count. Shadow orders (hypothetical one-lot orders placed at the back of the best bid and ask every second, never entering the book) measure bias in fill rate and 10-second markout P&L.

The pre-registered plan was frozen before any queue-model outcome was computed (but after my liquidity study analyzed the same days), with one confirmatory run; deviations, including a post-run shadow fill-rule change, are listed in the paper.

What I found

Over whole sessions, estimators miss true fill times by 0.31 seconds (risk_averse) to 1.07 seconds (front_optimistic) on average in crude, and 0.68 to 1.98 seconds in gas. The ranking misleads: owners cancel or move most tracked orders first (68% in crude), where the error is zero unless the estimator invents a fill, which risk_averse almost never does. On orders that fill, all six miss by 0.77 to 0.95 seconds in crude, and risk_averse is worst (post hoc).

Shadow fill-rate and P&L biases are small, and their sign depends on whether a true fill is read at the filled order's resting position (the plan's reading) or at the fill record's own price (post hoc). Neither metric alone says which estimator to use.

After the gas report, two estimators' errors are lower for orders joining the best quote in the next five minutes, relative to the same day's 09:00 to 09:55 baseline and to matched days (prob_log by 0.32 seconds, front_optimistic by 0.60), significant after adjusting for 12 tests. That mostly reflects which orders are scored, not better queue modeling: about 20 percentage points more of them fill, about five times faster, and the change within outcome type is not significant. In the same windows the shadow P&L bias of risk_averse and front_optimistic is larger, not smaller (plan's reading; a preliminary readout).

In crude, no estimator's error changed significantly after the petroleum report (-0.14 to +0.05 seconds), which is not the same as no effect: power was about 38% at a shift of 0.6 day standard deviations.

What it does not show

Shadows assume no market impact and zero latency; real orders' owners may be informed, so their error need not match a simulated strategy's. Every gas release day has the 08:30 ET jobless-claims release before the morning baseline; no control day does.

Reproduce it

The analysis code is private, and the CME market data are licensed and cannot be redistributed, so the results themselves cannot be rerun from public materials. The engine the study runs on, mbo-queue, is public (version 0.6.0 for the primary results). Its tests and a known-answer check on a synthetic CME-style exchange run from a clone:

The same per-estimator scoring pipeline also runs end to end on synthetic data, or on your own licensed whole-session Databento MBO file.

pip install "mbo-queue[data]"
git clone https://github.com/AnshDani2004/mbo-queue
cd mbo-queue
python tests/test_mboq.py
python scripts/synthetic_validation.py
python scripts/run_day.py --synthetic