Aug 2026 – present
mbo-queue
mbo-queue rebuilds a futures order book order by order from CME market-by-order (MBO) data, where every add, cancel, modify and fill carries an order ID. A backtest that sees only the total size at each price has to guess where a resting order sits in the queue. This library computes the actual place and measures how far the usual guesses miss.
What it computes
It applies CME's priority rules: new orders join the back of their price level, size increases and price changes lose priority, size decreases keep it, and a native iceberg refill goes to the back. On every fill it checks that the filled order is the one the rebuild has at the front; this FIFO-consistency rate must reach 95% before I trust any queue position.
For hypothetical shadow orders at the back of the best price, it runs six queue-position rules that see only aggregate depth and trade prints: hftbacktest's risk-averse rule (every cancel is behind you), its three power rules and its log rule, which follow Rigtorp's estimator, and a front-optimistic rule. Each is scored against the true queue on position error, fill rate, fill time and P&L, split into a fill-count part and an adverse-selection part.
CME data details
The book can only be rebuilt from Databento's 00:00 UTC snapshot, so the input must cover the whole session. MBO leaves out implied (spread-generated) liquidity and the hidden part of iceberg orders, so the ground truth is the displayed outright book. Its quality is measured (FIFO consistency, MBP-10 reconciliation), not assumed.
How I check it
One written spec (SPEC.md) has two implementations, a readable Python reference and a C++20 core with pybind11 bindings, and they return byte-identical output on randomized streams with every feature on. A synthetic CME-style exchange gives known answers: the risk-averse rule has zero error when cancels hit the tail of the queue, the front-optimistic rule when they hit the front, and with size-proportional cancels the linear rule's mean queue error is -0.022 lots against +1.769 for risk-averse (5 seeds). Scrambling every record after time T leaves earlier book samples unchanged. The suite has 53 tests.
Speed
On a synthetic stream of 229,313 records (one thread, Apple M3, version 0.6.0), the C++ core rebuilds the book, checks FIFO consistency and labels orders at 18.9M records per second, against 26.5k for the Python reference; with shadows and all six rules, 3.79M. Prebuilt wheels cover CPython 3.10 to 3.14 on Linux and macOS. The current release is 0.7.0, MIT-licensed.
What I used it for
It rebuilt the displayed order book of NYMEX crude oil (CL) and Henry Hub natural gas (NG) futures for my working paper, "Where does liquidity go around EIA releases?" (version 1, October 2026); the queue-model results are on the "How wrong are backtest queue models?" page. The repository contains no market data and no results computed from it.
Limits
Shadow orders never trade and change no one's behavior, the same assumption every replay backtester makes, so results apply to small orders. The rules follow Databento's CME Globex feed; other venues may need different ones.
Reproduce it
Install from PyPI; the data extra adds databento, pandas and pyarrow to read DBN files. The tests, the known-answer validation and the full pipeline on synthetic data run from a clone.
pip install mbo-queue
pip install "mbo-queue[data]"
git clone https://github.com/AnshDani2004/mbo-queue
cd mbo-queue
python tests/test_mboq.py
python scripts/synthetic_validation.py
python scripts/run_day.py --synthetic