Live options flow only matters if it is fast and complete. A feed that lags at the open, or silently drops prints during a burst, shows traders the wrong picture at exactly the wrong moment. In this post we walk through how we built a real-time options flow feed on Redis Streams and WebSockets for an options analytics platform, and how it keeps streaming smoothly during peak traffic.
The problem
Options traders watch live flow to spot large orders, sweeps and unusual activity the moment they hit the tape. That flow comes from the US options market’s consolidated feed (OPRA), one of the highest-volume public market data feeds, and its volume is anything but even. The market open, economic releases and sharp moves in popular underlyings produce bursts far above the daily average.
The platform’s previous approach struggled with those bursts. It dropped messages and stalled without telling anyone, so dashboards showed stale or incomplete flow when traders cared most. Our goal was a feed that keeps up at the open, never loses data silently and recovers on its own.
System architecture
The pipeline has five stages, each with one job:
- Ingestion. Python feed readers consume the vendor’s OPRA options feed and dark pool trades, parse and normalize each message, and hand it off immediately.
- Buffering. A bounded in-memory queue of 50,000 items sits between each reader and storage, so a slow write never blocks the feed.
- Storage. A writer drains the queue into Redis Streams in pipelined batches of up to 5,000 messages, with a separate connection pool per stream.
- Enrichment and aggregation. Trades are classified as buyer- or seller-initiated, and flow views such as premium by ticker and call and put activity are computed on a 60-second cycle across 8 parallel jobs.
- Delivery. Results are fanned out over WebSockets to the platform’s live analytics pages, which are served by more than 100 WebSocket routes.
Why we chose Redis Streams
Redis Streams gave us an ordered, persistent log for every feed with IDs consumers can resume from. That matters for live flow: if a worker restarts, it picks up where it left off instead of leaving a hole in the tape. Because Redis keeps the data in memory, writes and reads stay fast even at peak volume, and RediSearch lets us query live data for screeners without moving it into another database.
Handling peak traffic
- Explicit backpressure. The bounded queue forces a decision when storage falls behind. If it ever fills, the message is dropped, counted and alerted on, never lost silently. In normal operation the drop counter stays at zero.
- Batched, pipelined writes. Sending thousands of messages per round trip instead of one keeps writes ahead of the feed during bursts.
- Isolation between streams. A dedicated connection pool per stream means a surge in one feed cannot starve the others.
- Parallel aggregation. Partitioning the 60-second aggregation across 8 jobs keeps each job well inside its window, and a slow partition does not delay the rest.
Key engineering decisions
- Compact payloads. Each flow event carries only the fields the screen needs, which keeps WebSocket messages small and rendering fast.
- Bounded memory. Streams are trimmed by length and cache-style keys carry expiries, with memory limits and a deliberate eviction policy on a Redis store that holds more than 10 million keys.
- Accurate sentiment, then corrected sentiment. Trade side is inferred from the bid/ask spread first, with the tick rule as a fallback, and a correction pass five minutes later re-checks each trade once late data has arrived.
- Burst detection and alert cooldowns. Activity is counted in 200 ms buckets to catch sweeps and surges, and atomic cooldowns stop the same alert firing again for every print in a burst.
- Self-healing feeds. A watchdog tracks messages actually processed, not just whether the process is up, and restarts any stalled feed within three minutes.
What we learned
Real-time financial data systems reward obsessing over the unhappy paths: the burst at 9:30 a.m., the half-open socket, the queue that is almost full. Redis Streams proved to be the right foundation, fast and flexible enough to grow with the product, and the combination of Redis Streams and WebSockets is now the backbone of the platform’s live data.
Read the full real-time options flow and dark pool pipeline case study, or dig into the ingestion details in how we ingest the OPRA options firehose with Redis Streams. If you are building a trading or market data product, see our FinTech software development services.