Real-time options flow and dark pool pipeline
A market data pipeline that ingests the US options (OPRA) firehose and dark pool trades without dropping messages during market-open bursts.
- item bounded queue
- 50k
- messages per batch write
- 5,000
- burst detection buckets
- 200 ms
- automatic feed recovery
- < 3 min
The challenge
- Client
- Options analytics company (name withheld under NDA)
- Industry
- FinTech · Market data
Market open produces violent traffic bursts. The previous approach dropped messages and stalled silently, so dashboards showed stale or incomplete flow at the moments traders cared about most.
What we built
Stream writer with a bounded 50,000-item queue, a connection pool per stream, pipelined batch writes of 5,000 messages and alerts on any dropped message.
60-second aggregation across 8 parallel jobs feeding live dashboards over WebSockets.
Two-tier trade sentiment (bid/ask spread first, tick rule as fallback), a correction pass five minutes after each trade and atomic alert cooldowns.
Burst detection on 200 ms buckets and self-healing that restarts a stalled feed within three minutes.
How the system fits together
The path a request or message takes through the production system, end to end.
- OPRA firehose
- Bounded queue · 50k
- Redis Streams
- 60s aggregation × 8
- WebSocket fan-out
- Analytics UI