Cross Venue Price Discovery And Latency Aware Trading Engine

Problem

Cross-venue price-discovery studies often begin with already-cleaned bars or synchronized datasets. That hides many of the hardest questions: what was actually received, when it was received, whether timestamps from different systems are comparable, whether sequence gaps indicate missing subscribed messages, and whether repeated or unchanged feed messages represent corruption or normal exchange behavior.

This project asks what infrastructure and evidence are required before making a credible claim that one venue leads another.

The first goal is not to build a trading strategy. The first goal is to create an auditable path from exact public WebSocket frames to a validated analytical dataset.

Approach

I built concurrent public Coinbase and Kraken WebSocket collectors for BTC-USD trades and top-of-book updates.

Each exact raw frame is archived before decoding, together with its local receipt timestamp, connection epoch, source metadata, rotating JSONL shard, SHA-256 checksum, and session manifest.

A venue-aware quality layer evaluates archive integrity, parsing, timestamp behavior, identifier continuity, duplicate-message categories, quote freshness, connection liveness, and paired cross-venue overlap. Only accepted sessions can be promoted into validated dataset manifests.

A deterministic replay pipeline converts the validated raw evidence into exact-decimal Parquet trade, top-of-book, and raw-record outcome tables. DuckDB views provide an analytical interface without replacing the raw archives or Parquet files as the source of truth.

Current validated result

Three short paired calibration sessions were accepted under the fixed quality policy.

One accepted paired dataset was then used for the first complete deterministic normalization run.

The same validated input was normalized twice with matching event IDs, row counts, semantic hashes, Parquet checksums, and normalization manifests.

These figures describe one short accepted paired dataset, not the final multi-session research sample.

1. 5,361 raw archive records
2. 1,257 normalized trade rows
3. 3,803 normalized top-of-book rows
4. 5,361 raw-record outcome rows
5. Zero duplicate normalized event IDs

Technical pipeline

1. Concurrent public Coinbase and Kraken WebSocket collection
2. Immediate local receipt timestamping after every received frame
3. Exact raw-frame archival before JSON decoding
4. Rotating JSONL shards, SHA-256 checksums, and session manifests
5. Venue-aware quality analysis
6. Paired cross-venue overlap validation
7. Validated dataset manifests containing accepted sessions only
8. Deterministic replay through the existing venue parsers
9. Exact decimal128 trade and top-of-book Parquet schemas
10. One explicit normalization outcome for every raw record
11. Stable source and normalized event identifiers
12. Strict raw-to-normalized reconciliation
13. DuckDB analytical views over immutable Parquet files
14. Hash-chained campaign ledger and derived campaign registry
15. Fixed morning, afternoon, and evening collection campaign

Metrics

What failed or changed

The initial quality policy quarantined the first paired session even though the collectors, raw archives, and checksums were valid.

The investigation showed that several broad diagnostics were mixing expected exchange-feed behavior with genuine data-integrity risk.

Negative exchange-to-receipt timestamp differences on Coinbase and Kraken behaved like stable clock offsets. They could not be interpreted as measured one-way latency.

Coinbase product-level sequence jumps were too broad as a missing-data signal for the channels subscribed to by this project. Heartbeat and trade-identifier evidence provided more relevant continuity diagnostics.

Kraken's aggregate exact-duplicate rate was dominated by repeated heartbeat and ticker frames rather than duplicated or conflicting trades.

Quote age alone also treated an unchanged best bid and offer as a feed failure, even while heartbeats and other messages continued arriving.

The policy was recalibrated using record-level evidence, documented venue semantics, regression tests, and before-and-after dispositions. The observations were retained, but normal feed behavior no longer automatically quarantined a session.

What improved

The calibrated quality policy separates stable timestamp offsets from unstable timestamp behavior, product-level sequence diagnostics from missing-trade evidence, control-frame duplication from duplicate trades, and quote age from connection inactivity.

All three subsequent paired calibration sessions were accepted with valid archives, zero parse errors, zero reconnects, sufficient overlap, and successful promotion dry runs.

The normalization layer then demonstrated complete source reconciliation and deterministic output on an accepted paired dataset.

Current phase

The project is now running a fixed multi-session collection campaign.

The campaign records every scheduled attempt, including accepted, quarantined, rejected, failed, aborted, and missed sessions. Inclusion is governed only by predeclared quality rules, so accepted sessions cannot be manually removed after seeing later research results.

The target is at least ten accepted paired sessions, five accepted hours of overlap, three local calendar dates, and morning, afternoon, and evening coverage.

The campaign is still in progress. No exploratory price-discovery conclusion has been produced.

Final interpretation

The strongest result so far is an engineering and research-design result.

The system can collect simultaneous public data from two venues, preserve the exact raw evidence, distinguish expected feed behavior from integrity failures, promote accepted sessions, and deterministically reproduce analytical datasets with complete source lineage.

This does not yet show that Coinbase leads Kraken, that Kraken leads Coinbase, or that a tradeable latency opportunity exists.

Those questions require the broader multi-session dataset and later statistical research phases.

Limitations

• The completed normalization result currently covers one short accepted paired dataset.
• The three accepted calibration sessions were collected sequentially and do not establish broad time-of-day or market-regime coverage.
• The broader multi-session campaign is still underway.
• Exchange timestamps and local receipt timestamps are preserved but not synchronized or corrected.
• Exchange-to-receipt timestamp differences are not treated as measured one-way network latency.
• Exact duplicate raw frames are preserved rather than removed.
• The current system covers public Coinbase and Kraken BTC-USD feeds only.
• No formal lead-lag analysis has been completed.
• No predictive model, backtest, execution simulation, PnL result, or live trading system exists.
• The source repository is private during active development.

Next steps

1. Complete the fixed multi-session collection campaign
2. Finalize the validated campaign manifest
3. Normalize the complete multi-session dataset
4. Produce exploratory spread, quote-duration, trade-intensity, and data-coverage diagnostics
5. Test formal event-time and lead-lag hypotheses
6. Evaluate stability across sessions and market conditions
7. Add predictive or latency-aware simulation only when supported by the evidence

How to reproduce

The private repository contains typed configurations, deterministic commands, source checksums, raw and normalized manifests, complete raw-record reconciliation, automated tests, continuous-integration checks, and audit reports for each completed phase.

This project page reports only results that passed the repository's validation gates.

The repository remains private during active development, so the current project cannot yet be independently reproduced by an external reader.

repository: private during active development