What is conflation in a market data system?

Conflation is coalescing many updates within a time window down to a single one — keeping only the latest value per symbol and emitting it on a fixed cadence, rather than forwarding every tick. CME’s market-data platform defines it exactly: “Conflated market data combines multiple updates within an interval to a single event.” It is the central idea behind a trading dashboard because the underlying feed moves far faster than any screen can use — a single hot symbol can tick thousands of times a second, while a human eye and a monitor can only use a handful of updates a second. So for each symbol the system keeps one slot holding the latest price; every incoming tick overwrites it; and on a cadence (often every 250 ms, i.e. 4 a second) it emits whatever is in the slot. Every tick dropped between emits was already stale. This is safe precisely because a price is a last-writer-wins value, not an event you must not miss — dropping the intermediate ticks is the feature, not a bug, and it means a conflation buffer can never build an unbounded backlog because there is only ever one slot per symbol.

How does a trading dashboard scale to a million concurrent viewers?

By conflating first, then fanning out one update per symbol over a WebSocket gateway fleet. Naively forwarding every tick to every viewer is impossible: 1M viewers × 30 on-screen symbols is 30M live subscriptions, and at ~200 ticks/s per symbol × 40 bytes that is roughly 240 GB/s of egress. Conflation caps each symbol at the display rate (say 4/s), so the same 30M subscriptions cost 30M × 4 × 40 B ≈ 4.8 GB/s — a ~50× cut on average and over 1,000× on the hottest symbols. The conflated updates go onto a pub/sub bus keyed by symbol, and a fleet of WebSocket gateways subscribes to it; each gateway holds ~50,000 client sockets and pushes an update only to the sockets that subscribed that symbol. A hot symbol watched by 300,000 people is still one message on the bus — the multiplication happens at the edge, in the gateways. The gateway count is therefore sized by open connections (~1M ÷ 50k ≈ 20 boxes), not by data rate.

Why send a snapshot before deltas when a client subscribes?

Because a delta — “price up 2¢” — is meaningless without a base to apply it to. So on subscribe the gateway first sends a snapshot: the full current quote, read cheaply from the symbol’s latest conflation slot and stamped with a sequence number. The client paints immediately, then receives only numbered deltas from there. The sequence numbers matter twice: they let a client detect a dropped delta (it sees seq 103 then 105 and knows 104 is missing) rather than silently rendering a wrong price, and the cure for any gap is not to chase the lost delta but to request a fresh snapshot and reset — one round-trip repairs any amount of missed history. This is exactly how real exchange feeds like CME’s MDP 3.0 work: an incremental refresh stream plus a snapshot channel for recovery. The same snapshot mechanism onboards a brand-new tab and heals a client that fell behind.

How does a trading dashboard handle a slow client without falling over?

By conflating again per-connection instead of queuing a backlog. With a million clients there is always a fat tail of slow ones — a phone on a train, a laptop with forty tabs — and if the server queues undelivered updates for a client that can’t keep up, one slow socket becomes a memory leak that stalls a gateway serving fifty thousand healthy clients. The fix is to apply the same last-writer-wins idea at the socket: the per-connection buffer holds one pending frame per symbol and overwrites it, so a slow client receives fewer updates but always the current price, never a stale backlog. Market-data vendors call this just-in-time conflation — protecting a client whose bandwidth or CPU can’t keep up. Conflation is therefore not one component but a discipline applied at every layer (at ingest per symbol, on the bus, and per client socket), so no layer can ever build an unbounded queue and back-pressure always means “send less,” never “stall the producer.”

How is portfolio P&L computed in real time?

Unrealized P&L is a join: take the user’s positions — quantity and average cost — and compute (last − cost) × qty for each, recomputed every time the price ticks. The architectural insight is that it is the live price stream fanned onto every holder of a symbol, so the natural home for the multiply is the client itself: the browser already subscribes to its portfolio’s symbols for the display and multiplies locally, so the server sends one conflated price and ten thousand clients holding that symbol each do their own trivial arithmetic — which stops P&L from becoming a server-side fan-out nightmare. Crucially the two inputs come from different lanes: the live price is the lossy display lane (conflated, best-effort), but the positions and cost basis come from the exact lane — the authoritative order/book store, updated only on real fills. Marking an exact position to a slightly-stale display price is fine (the P&L may flicker by a cent), but the position quantity itself must never be approximate.

Design a Trading Dashboard

A million people watching prices that move millions of times a second, and a browser that can render maybe ten of them. Built from zero: the read-side scope (the display, not the matching engine), a commit-first envelope where naively forwarding every tick is ≈240 GB/s and conflation cuts it to ≈4.8, market-data fan-in and normalization across venues, then the signature idea — conflation, keeping one latest slot per symbol and emitting ~4 times a second (exactly what Interactive Brokers ships), pub/sub fan-out over a WebSocket gateway fleet, snapshot-then-delta on subscribe with sequence-gap resync, per-client backpressure that conflates again at the socket, tiered update rates that follow attention, OHLC candle roll-ups computed on the full tick stream, portfolio P&L as the live price joined onto positions, and the two-lane guarantee split — lossy display, exact books — that runs through the whole design.

System design · Systems. The source ↗

A free, interactive, animated visual explainer of Design a Trading Dashboard — built to be understood, not skimmed.

Questions

What is conflation in a market data system?: Conflation is coalescing many updates within a time window down to a single one — keeping only the latest value per symbol and emitting it on a fixed cadence, rather than forwarding every tick. CME’s market-data platform defines it exactly: “Conflated market data combines multiple updates within an interval to a single event.” It is the central idea behind a trading dashboard because the underlying feed moves far faster than any screen can use — a single hot symbol can tick thousands of times a second, while a human eye and a monitor can only use a handful of updates a second. So for each symbol the system keeps one slot holding the latest price; every incoming tick overwrites it; and on a cadence (often every 250 ms, i.e. 4 a second) it emits whatever is in the slot. Every tick dropped between emits was already stale. This is safe precisely because a price is a last-writer-wins value, not an event you must not miss — dropping the intermediate ticks is the feature, not a bug, and it means a conflation buffer can never build an unbounded backlog because there is only ever one slot per symbol.
How does a trading dashboard scale to a million concurrent viewers?: By conflating first, then fanning out one update per symbol over a WebSocket gateway fleet. Naively forwarding every tick to every viewer is impossible: 1M viewers × 30 on-screen symbols is 30M live subscriptions, and at ~200 ticks/s per symbol × 40 bytes that is roughly 240 GB/s of egress. Conflation caps each symbol at the display rate (say 4/s), so the same 30M subscriptions cost 30M × 4 × 40 B ≈ 4.8 GB/s — a ~50× cut on average and over 1,000× on the hottest symbols. The conflated updates go onto a pub/sub bus keyed by symbol, and a fleet of WebSocket gateways subscribes to it; each gateway holds ~50,000 client sockets and pushes an update only to the sockets that subscribed that symbol. A hot symbol watched by 300,000 people is still one message on the bus — the multiplication happens at the edge, in the gateways. The gateway count is therefore sized by open connections (~1M ÷ 50k ≈ 20 boxes), not by data rate.
Why send a snapshot before deltas when a client subscribes?: Because a delta — “price up 2¢” — is meaningless without a base to apply it to. So on subscribe the gateway first sends a snapshot: the full current quote, read cheaply from the symbol’s latest conflation slot and stamped with a sequence number. The client paints immediately, then receives only numbered deltas from there. The sequence numbers matter twice: they let a client detect a dropped delta (it sees seq 103 then 105 and knows 104 is missing) rather than silently rendering a wrong price, and the cure for any gap is not to chase the lost delta but to request a fresh snapshot and reset — one round-trip repairs any amount of missed history. This is exactly how real exchange feeds like CME’s MDP 3.0 work: an incremental refresh stream plus a snapshot channel for recovery. The same snapshot mechanism onboards a brand-new tab and heals a client that fell behind.
How does a trading dashboard handle a slow client without falling over?: By conflating again per-connection instead of queuing a backlog. With a million clients there is always a fat tail of slow ones — a phone on a train, a laptop with forty tabs — and if the server queues undelivered updates for a client that can’t keep up, one slow socket becomes a memory leak that stalls a gateway serving fifty thousand healthy clients. The fix is to apply the same last-writer-wins idea at the socket: the per-connection buffer holds one pending frame per symbol and overwrites it, so a slow client receives fewer updates but always the current price, never a stale backlog. Market-data vendors call this just-in-time conflation — protecting a client whose bandwidth or CPU can’t keep up. Conflation is therefore not one component but a discipline applied at every layer (at ingest per symbol, on the bus, and per client socket), so no layer can ever build an unbounded queue and back-pressure always means “send less,” never “stall the producer.”
How is portfolio P&L computed in real time?: Unrealized P&L is a join: take the user’s positions — quantity and average cost — and compute (last − cost) × qty for each, recomputed every time the price ticks. The architectural insight is that it is the live price stream fanned onto every holder of a symbol, so the natural home for the multiply is the client itself: the browser already subscribes to its portfolio’s symbols for the display and multiplies locally, so the server sends one conflated price and ten thousand clients holding that symbol each do their own trivial arithmetic — which stops P&L from becoming a server-side fan-out nightmare. Crucially the two inputs come from different lanes: the live price is the lossy display lane (conflated, best-effort), but the positions and cost basis come from the exact lane — the authoritative order/book store, updated only on real fills. Marking an exact position to a slightly-stale display price is fine (the P&L may flicker by a cent), but the position quantity itself must never be approximate.

Design a Trading Dashboard

Questions

Related explainers