R21 — High-volume data flow: fan-out and fan-in — Instagram, live scores, notifications, Datadog

R20 moved a few very large objects. This section moves the opposite: billions of tiny events — a post, a like, a metric point, a log line — each only a few hundred bytes, but arriving at hundreds of thousands to hundreds of millions a second. The hard part is not storing any one of them. It is the shape of the flow: one event that must reach millions of readers, or millions of writers pouring into one store.

The format is the same as before: the shared problem and its toolkit first, then four walkthroughs — three kinds of fan-out, and fan-in as their mirror image.

The shared concern — which way the data flows

There are two shapes:

• Fan-out — one write, many readers. A user with ten million followers posts once; ten million home feeds must show it. Reads vastly outnumber writes, and each write can multiply into millions of pieces of work.
• Fan-in — many writers, one destination. A million servers each send metrics every ten seconds into one monitoring platform. Writes vastly outnumber reads, and the destination must absorb them without falling over.

Five problems follow, each from the one before.

Problem 1 — decide when to do the work

For fan-out, you either do the work on write — copy each post into every follower's feed as it is created (push) — or on read — gather posts from everyone a user follows when they open the app (pull). Push makes reads cheap and writes expensive; pull does the reverse.

For fan-in the same choice appears as aggregate on ingest — combine points as they arrive — or store raw and aggregate at query time. Ingest-time aggregation is cheap to query and loses detail; raw storage keeps everything and is expensive.

Problem 2 — absorb bursts with a log

Neither shape arrives smoothly. A celebrity posts; a data centre restarts and every host reports at once. Put a durable log such as Kafka (R9) between the writers and the work. It absorbs bursts, lets each consumer go at its own pace, and lets you replay after a bug. The writer is acknowledged as soon as the event is safely in the log, not when all the work is done.

Problem 3 — skew breaks the average

Averages hide the hard case. Most users have a few hundred followers; a few have a hundred million, so pushing their posts would mean a hundred million writes each. Most metrics have a handful of tag combinations; one careless tag such as a user id creates millions of distinct series. Designs for high-volume flow are designs for the skewed tail, not the average. The usual answer is a hybrid: push for the many small cases, and handle the few huge ones differently.

Problem 4 — reduce before you store

At this volume you cannot keep every event in its most expensive form. Fan-out stores only ids in feeds and fetches the content once, from cache. Fan-in pre-aggregates — sums, counts, minimums and maximums per ten seconds, at the source — compresses (Datadog-style time series reach about two bytes a point), and rolls old data up into coarser intervals. For counts and distributions it uses mergeable sketches, such as HyperLogLog for distinct counts, which can be combined across machines without losing correctness.

Problem 5 — choose freshness per use

Not every reader needs every event instantly. A like count can lag a few seconds; an alert on a production outage cannot. Decide freshness per use, and spend money only where it is needed.

Rules of thumb for high-volume data flow

• Name the shape first: fan-out or fan-in, and the ratio of reads to writes.
• Choose per case whether work happens on write or on read — and use a hybrid for the skewed tail.
• Put a durable log between producers and the work; acknowledge once the event is in the log.
• Store ids, aggregates and sketches, not full copies of everything.
• Decide freshness per use; not every number needs to be live.

How the four walkthroughs build on each other

They are ordered so that each adds one difficulty, and the last mirrors the rest:

1. Instagram — fan-out. Every choice is about making the read cheap: precomputed timelines for most users, celebrities merged in at read time, likes and counts kept approximate, and media off the request path using R20's toolkit.
2. Live scores — the most extreme fan-out: one tiny update, from one writer, to tens of millions of screens within seconds, where every update supersedes the last. Push through an edge fan-out tree, conflate, and let late joiners start from a snapshot.
3. Notifications — fan-out to people who are not in the app, through providers we do not control, each with their own preferences and quiet hours. Priority lanes, deduplication, rate limits and provider failover become the design.
4. Datadog — fan-in, the same problems seen from the other side. Every choice is about making the write cheap: aggregation at the source, a buffer that takes the acknowledgement, compression and rollups, cardinality as the real cost, and an alerting system that has to tell "the customer's host is down" apart from "our own pipeline is slow".

At a glance

Walkthrough 1 — Instagram, a read-heavy media feed

Where this starts: fan-out. One post must appear in many feeds, and every design choice is about making the read cheap without letting a celebrity's post turn into a hundred million writes. Media uploads reuse R20's toolkit — pre-signed uploads, async transforms, a CDN — so the interesting part is the feed.

Scope and requirements

Interviewer: Design Instagram.

You: Each feature here is its own sub-system, so I want to narrow it first. The questions I'd ask are the ones whose answers move boxes: which features are in — posts, feed, stories, likes, follows, explore, DMs? Photos only, or video too? Ranked or chronological feed? How fresh must the feed be after someone posts? Are there accounts with tens of millions of followers? Private accounts and blocking? Global or one region?

Interviewer: Assume whatever is reasonable.

You: Then I'll write these down. Photo and short-video posts (up to 60 s), a ranked home feed, follow and unfollow with private accounts, likes and comments, stories that expire after 24 hours, notifications, and search and explore at a high level. 500M DAU, 2B registered, global. 100M posts a day, 80% photo, and 500M stories a day. Some accounts have 100M+ followers, and reads outnumber writes about 100 to 1. The feed may lag a new post by a few seconds and like counts may be approximate — that one sentence is what lets most of the work go asynchronous.

Non-functional: 99.95% for feed reads and uploads; feed p99 under 300 ms server-side; media first byte under 200 ms from the CDN; uploaded media never lost; 3x peak headroom plus viral spikes on single posts; and cost, because media storage and egress dominate the bill. Out of scope: DMs, live video, ads, shopping, moderation models, auth internals.

What's being probed: whether you scope before designing, and whether you spot the questions that change the architecture.

Concept — Architecture-shaping questions. A clarifying question earns its time only if a different answer gives a different design. "Are there celebrities?" decides between push and pull feeds. "Is a few seconds of delay fine?" decides whether work can be asynchronous. "Video or photos?" decides whether you need a transcoding pipeline. When the interviewer says "assume whatever", say the assumptions aloud and write them down, because every later number depends on them.

Takeaway: Ask the questions whose answers would change the boxes on the whiteboard.

Interviewer: You said prefer availability. So everything is eventually consistent?

You: No, I choose per data type. Feeds, counts, search and notifications are eventual; nobody notices a like count one second behind. Three narrow paths need more. Username uniqueness, because two people cannot claim one handle. Read-your-writes on my own post and follow, because after I post I must see it on my profile at once. And privacy and block checks, because a stale replica must never show my content to someone I blocked. R0 covers CAP; read-your-writes is the practical refinement — route my reads of my own data to the primary. AP for the feed and counters, strong on the paths where a user would notice or be harmed.

Capacity

Interviewer: Size it.

You: Assumptions: 10 feed opens per user per day, 20 posts per page, 5B likes a day, 500M comments, 200M follows, 3x peak. A stored photo with four sizes is about 3 MB, a video with renditions about 30 MB, about 200 KB of media served per feed item.

The number that matters is the last one: 1.2K posts a second become 230K feed inserts a second. That is why fan-out must be asynchronous and batched.

Storage: photos 240 TB a day, video 600 TB a day, so ~840 TB a day and ~1.5 EB over five years before tiering. Stories are ~1 PB a day but only 24 hours stays hot. Post metadata is 100 GB a day, ~180 TB over five years. Likes ~450 TB over five years. The follow graph, stored in both directions, is ~24 TB. Bandwidth: ingress ~78 Gbps for posts plus ~93 Gbps for stories. Egress is 5B loads x 20 items x 200 KB = 20 PB a day, ~1.85 Tbps average and ~5.5 Tbps peak, and over 95% of it must come from CDN edges.

Cache: 500 post IDs per active user is 2 TB raw, ~5 TB with Redis sorted-set overhead, about 100 nodes with replicas. Hot post metadata ~200 GB, counters ~6.4 GB, story tray ~50 GB. So metadata is small, media is huge, and egress is the cost problem.

What's being probed: whether you can do quick math and notice which number drives the design.

Concept — Write amplification. One logical write that causes many physical writes. In a push feed, one post becomes one insert per follower, so a harmless-looking write rate becomes the largest load in the system. Estimation is not about precise numbers; it is about finding the term that dominates — here the multiplier, not the base rate. Storing 8-byte IDs rather than posts in the timeline is what keeps the feed cache small enough to hold for every active user.

Takeaway: Always multiply by the fan-out factor; the small number times a big multiplier is your real load.

API

Interviewer: Sketch the APIs. How does upload work?

You: Everything goes through the gateway with an OAuth bearer token. The main surface:

Upload is three steps: ask for a slot, PUT the bytes straight to object storage, then create the post.

POST /v1/media/uploads
{ "type": "image/jpeg", "size_bytes": 3145728, "sha256": "9f2c..." }
200 OK
{ "media_id": "2873459812734591234", "upload_url": "https://media-upload.example.com/raw/2873...?X-Amz-Signature=...", "expires_in": 900 }

POST /v1/posts
Idempotency-Key: 7d1e0c4a-5b8f-4f8e-9a21-0e6b1c2d3f44
{ "media_ids": ["2873459812734591234"], "caption": "Sunset #goa", "location_id": 1123 }
202 Accepted
{ "post_id": "2873460000123456789", "status": "PROCESSING" }

It returns 202 because the media is still being processed. A retried post with the same key returns the original post_id — that is R3's key table with a 24-hour TTL. Pagination is a cursor encoding the last score and post ID, because offsets on a live feed show you posts you have already seen when new ones land at the top (R11 has the rows-read argument). Rate limits per user and IP at the gateway stop like-bots (R14).

What's being probed: whether your API reflects the real workflow, including async processing and large payloads.

Concept — Control plane and data plane. The control plane is small requests that decide things: who may upload, what the post says. The data plane is the heavy bytes. Mixing them makes API servers scale with megabytes instead of requests. A pre-signed URL is a time-limited signed link that lets the client write one object straight to storage without holding storage credentials, and 202 Accepted is the honest response for work that finishes later.

Takeaway: Let the API decide and the storage carry the bytes.

Data model

Interviewer: Main entities, and where each lives?

You:

Interviewer: Why not put everything in Cassandra and run one database?

You: I'd consider it with a smaller team. Three things keep users, posts and follows relational: I want single-user transactions (create the post and bump the post count together); username uniqueness is a plain unique constraint rather than Cassandra lightweight transactions; and the hot queries are single-shard range scans PostgreSQL does well. Let me list them, because the store falls out of them:

• Feed page: ZREVRANGEBYSCORE feed:{uid} → post_ids → multi-get metadata → multi-get counters → liked_by_me.
• Profile grid: SELECT … FROM posts WHERE author_id = ? AND post_id < ? ORDER BY post_id DESC LIMIT 12 on one shard.
• Followers of X for fan-out: scan the followers partition in pages of 1,000.
• Whom X follows, for pull and the story tray: read the following partition.

Likes, comments and notifications are the opposite — billions of tiny appends with no joins — which is Cassandra's sweet spot. Instagram has publicly described both sharded PostgreSQL and Cassandra.

What's being probed: whether you choose stores from access patterns rather than habit, and can defend the choice under pressure.

Concept — Access-pattern-first modelling. R10 covers choosing stores; the interview move is to say the four or five hot queries out loud before naming a database. Relational wins when you need transactions, constraints and flexible queries at modest write rates. Wide-column wins at enormous append rates on known queries keyed by one partition. Every extra store must earn its place with a workload the others handle badly.

Takeaway: Write down the hot queries first; the store choice usually falls out of them.

High-level design

Interviewer: Draw it.

You: Media bytes go client to object storage directly. Everything else goes through the gateway. Kafka decouples a write from its side effects — fan-out, notifications, search indexing, counters, analytics.

Click through the diagram's flows — upload, fan-out, feed read — and watch where the bytes go versus where the decisions go: the services never touch media on the read path.

The boxes and their one job each: the CDN caches every media variant with long TTLs because media is immutable. The gateway does TLS, auth, rate limiting, routing. The upload/post service issues pre-signed URLs, writes the post as PROCESSING, flips it to PUBLISHED when media is ready and emits PostPublished. Media workers resize images (150, 320, 640, 1080 px), strip EXIF, transcode video to HLS/DASH, run safety classifiers, autoscaled on queue depth. Fan-out workers page through followers and push post_ids into Redis timelines. The feed service merges, ranks and hydrates. The social graph service owns follows, blocks and privacy. The engagement service owns likes and comments. Notification and search consume events.

Interviewer: Walk me through opening the app.

You: The feed service reads ~500 candidate IDs from feed:{uid}, asks the graph service which celebrities I follow and pulls their recent post IDs from per-author caches, filters seen, blocked and deleted posts, and sends ~300 candidates to ranking. It takes the top 20 and hydrates them — batched multi-gets for metadata, counters and liked_by_me. The response carries CDN URLs sized for the device. If my Redis timeline is missing because I was inactive, it is rebuilt by pull. Store IDs in timelines and hydrate at read time, so an edited caption or a new count is always fresh and stale IDs are harmless.

Deep dive — the upload pipeline

Interviewer: 100M uploads a day, some of them hundreds of MB. Defend the design.

You: The client gets a 15-minute pre-signed PUT, uploads to the raw/ bucket — multipart and resumable for video, compressed client-side first, with a sha256 for dedupe and integrity. It creates the post with an idempotency key; the row lands as PROCESSING on the author's shard. Object storage emits object-created onto a queue; stateless workers produce variants into media/, with backoff retries and a DLQ for poison files (R9). MediaReady flips the post to PUBLISHED and emits PostPublished; fan-out, indexing and tag notifications follow. The author sees the post at once, read from the primary.

Proxying bytes through app servers would make the app tier scale with megabytes and hold slow uploads open; going direct costs only post-upload validation and orphan cleanup. Two more details. Media keys are immutable and content-addressed, so CDN TTLs can be effectively forever and invalidation never arises (R12). And lifecycle rules delete raw/ objects with no post after 24 hours and move old originals to colder tiers.

Deep dive — the home feed

Interviewer: Push or pull?

You:

Reads outnumber writes 100 to 1, so I want reads cheap, which favours push — but push breaks for celebrities. So hybrid. Authors above a threshold, say 100K followers and tuned by measurement, are flagged; their posts go only to a per-author recent-posts cache and are merged at read time. I push only to followers active in the last ~30 days; inactive users rebuild by pull on login. Fan-out runs in batches of 1,000 followers per task across Kafka partitions, so a 90K-follower author becomes 90 parallel tasks doing ZADD feed:{follower} <ts> <post_id> and trimming to 500. Redis loss is a cache miss — rebuild from the Cassandra copy or by pull.

What's being probed: the core feed trade-off, balanced with the read/write ratio.

Concept — Hybrid fan-out. Push is the mail carrier putting a copy in every mailbox: reading is instant, sending is expensive for a big list. Pull is everyone walking to a notice board: sending is free, every reader pays each time. Which is cheaper depends on the read/write ratio and how skewed audience sizes are, so real systems decide per author by follower count. When an account crosses the line, stop pushing and let old pushed entries age out — no migration.

Takeaway: Push for the many small authors, pull for the few huge ones.

Interviewer: A 100M-follower celebrity posts and 20M of those followers open the app in the same minute.

You: The write is cheap now — one row and one cache entry. The risk has moved to the read side: 20M feed loads reading the same key is a hot key, not a fan-out problem. A small in-process LRU on each feed host for celebrity post lists and hot metadata, refreshed every second or two, so most reads never leave the host. Request coalescing so one miss refills while the rest wait. Replicas of the hot key if needed. The CDN serves the media behind an origin shield, and the like counter switches to sharded mode. R12 covers each of these fixes; the point to say is that moving work from write time to read time trades a fan-out storm for a hot key.

Interviewer: And ranking inside 300 ms?

You: A funnel. Candidate generation (~500 pushed and pulled), a cheap filter (blocked, deleted, seen, low quality), a model predicting like, comment, save, share and dwell from precomputed features in a feature store, then re-ranking for diversity and inserted suggestions and ads. The ranked page is cached briefly so pagination is stable. The expensive model only scores a few hundred items. If ranking is slow, fall back to chronological from the same candidates.

Deep dive — follows, likes, stories, IDs

Interviewer: How do you store follows? Fan-out needs "who follows X", the tray needs "who does X follow".

You: Store each edge twice: following partitioned by follower, followers by followee, so both directions are single-partition scans and "does A follow B" is a point lookup. The two writes land on different shards, so it is R4's outbox: write following plus an outbox row locally, a relay publishes, an idempotent consumer writes followers, and a reconciler repairs drift (R6). Counts are denormalised and updated asynchronously. A 300M-follower partition is split by (followee_id, bucket). On UserFollowed, a worker backfills the followee's last ~20 posts into my timeline so the new follow shows up.

Interviewer: A post goes viral at 50K likes a second.

The trap: UPDATE posts SET likes = likes + 1

You: Every like locks the same row. The like edge (post_id, user_id) is the source of truth and dedupes double likes. Normal posts use INCR likes:{post}. Posts detected as hot switch to N sub-keys; writers pick one at random, readers sum and cache the total for a second or two. Kafka consumers batch increments per post per second into the durable table, and the UI shows "1.2M likes".

onLike(postId, userId):
  if cassandra.insertIfNotExists(likes, (postId, userId)):
      shard = isHot(postId) ? random(0, N-1) : 0
      redis.incr("likes:" + postId + ":" + shard)
      kafka.send("post-liked", postId, {postId, userId, ts})

Interviewer: Redis INCR is atomic and does 100K ops a second. Why shard at all?

You: Atomic is not the same as scalable. One key lives on one mostly single-threaded shard. A viral post at 50K/s plus reads plus every other key on that node can saturate it and slow every user who shares the node. Sharding spreads writes and memory across N nodes, and I only do it for posts detected as hot.

What's being probed: contention, and whether you separate the source of truth from the derived value.

Concept — Sharded counters. One tally sheet a whole stadium writes on becomes N sheets; each writer picks one, a reader adds them up, and the sum is cached briefly. R11 shows the same move as the hot-row demo. Because the count is derived from the like edges, it can always be rebuilt, which is exactly why it is safe for it to lag or be approximate.

Takeaway: Store the facts exactly; let the totals be sharded, batched and approximate.

Interviewer: Stories, and post IDs across thousands of shards?

You: Stories write to Cassandra USING TTL 86400 and object storage with a ~25-hour lifecycle rule, slightly longer so media never disappears while its metadata is still visible. The tray is a pull: read my followees, check active_stories:{author} in Redis, rank unseen first; celebrities stay pulled, and pushing a tray entry is the alternative if users follow thousands. Viewer lists and seen-state carry the same TTL. If data has an expiry date, let every layer delete it for you.

For IDs, Instagram publicly described generating them inside PostgreSQL per logical shard — R0 has the Snowflake family:

Time-sortable, so feed order and cursors use the ID directly; no ID service; and the shard inside the ID routes a lookup with no directory.

What's being probed: whether you planned for growth from day one.

Concept — Logical versus physical shards. user_id % servers moves most data when you add a server. Instead create thousands of logical shards up front and map them to a handful of physical hosts with a small table. Growth means copying a logical shard to a new host and flipping the mapping; IDs never change because they embed the logical shard, not the host. R10 covers this alongside consistent hashing.

Takeaway: Over-provision logical shards early so resharding is a move, not a re-key.

Failures and scale

Interviewer: Where does it break first?

You:

The rebuild-on-miss path is the same code that serves inactive users, so the recovery path runs every day rather than only during an outage. If a cache can't be rebuilt from durable data, it is a database.

Interviewer: A whole region goes down.

You: Reads are active-active; each user has a home region for writes to their shard, with async replicas elsewhere (R13). Media replicates cross-region asynchronously; the CDN is global. Cassandra runs multi-DC at LOCAL_QUORUM. On failure, DNS shifts traffic and replicas are promoted. RPO is seconds for metadata and near zero for media already on multi-AZ storage. I'd say the trade aloud: users homed in the failed region may lose their last few seconds of writes, in exchange for not paying cross-region latency on every write.

Curveballs

Interviewer: I follow someone new and don't see their posts. And unfollow, block, delete?

You: The first is fan-out that happened before the follow; the UserFollowed backfill fixes it, and celebrities need nothing because they are pulled. For the rest, remove the edge or soft-delete the post and emit an event. Hydration filters against the current truth, so correctness never depends on cleaning every cache; background removal is an optimisation. Deleted media becomes unreachable once metadata is gone, with a CDN purge for compliance.

Interviewer: "Liked by me" for 20 posts? Right image size per device? Notification floods? Explore?

You: A batched lookup on user_likes, with a per-user Redis set of recent likes and optionally a Bloom filter for fast negatives (R10). A fixed set of widths, with the API picking the best fit for device width and density — higher CDN hit rate than resizing at the edge. And aggregation by (recipient, post, type) in a window — "alice and 5,000 others liked your post" — with per-user push limits. Explore is offline candidate pools from embeddings, then online filter, rank and diversify; search is Elasticsearch with edge n-grams, ranked by prefix, follower count and affinity, fed from Kafka.

Quick reference

The 60-second version

Clients upload media straight to object storage with pre-signed URLs; async workers resize and transcode; a CDN serves immutable variants and carries nearly all of the ~2 Tbps egress. Users, posts and follows live in sharded PostgreSQL with thousands of logical shards and time-plus-shard IDs. Likes, comments, stories and notifications live in Cassandra, stories with a native 24-hour TTL. The feed is hybrid: post IDs pushed into Redis timelines for normal authors and active followers, celebrities pulled at read time, ~500 candidates ranked in two stages and hydrated in batches. Follows are stored in both directions and kept in sync by an outbox. Likes dedupe on the edge, and hot counts are sharded and approximate. Kafka decouples every write from its side effects, and every cache is rebuildable.

Walkthrough 2 — Live scores: one ball, fifty million screens

Building on Instagram: Instagram's fan-out was one post to many feeds, either copied into each feed when posted or gathered when the reader opened the app. Here the shape is more extreme and simpler at the same time. One tiny update — "four runs, 172 for 3 after 17.4 overs" — must reach tens of millions of screens within a few seconds, and every update supersedes the one before it. Nobody needs an old score. That one property changes the toolkit: we stop thinking about storing copies per reader and start thinking about pushing the latest value through a tree of servers, and letting slow readers skip straight to now.

Scope and requirements

Interviewer: Design a live cricket score service, something like Cricinfo. Think of an India match in a World Cup final.

You: The core is getting each ball's result to a huge audience quickly and in the right order. A few questions first:

Interviewer: Our own scorers enter each ball on a tablet at the ground or from the broadcast. Fresh within about five seconds is acceptable; faster is better. Plan for 50 million concurrent viewers on one match, with a handful of other matches live at the same time. Apps, web and some partner widgets. Text commentary yes, clips as links, career stats yes. Alerts for key events. And yes, scorers do get balls wrong and fix them.

You: Requirements, then.

Functional:

• A live view per match: score, overs, current batters and bowler, the last few balls, run rate and required rate.
• A full scorecard: every batter's and bowler's figures, partnerships, fall of wickets, extras.
• Ball-by-ball text commentary, newest first, with older balls loaded on scroll.
• A list of live and upcoming matches.
• Corrections: a wrongly entered ball can be fixed, and everyone's screen follows.
• Key events — a wicket, a fifty, the result — trigger alerts to subscribed users.
• Historical and career statistics, updated from the match data.

Non-functional:

• Ordered and never backwards on screen, except for an explicit correction.
• A new ball visible to 95% of viewers within five seconds of the scorer entering it; aim for two.
• 50M concurrent viewers on one match, with spikes when the match gets close.
• The scorecard always equals what the ball events add up to — no drift.
• The score stays readable even if the real-time layer fails: degrade to slower updates, never to a blank page.
• Cost matters: the audience is huge and mostly free.

Each update replaces the last, so a viewer needs only the latest state, never the history. That is what makes fifty million viewers affordable: we can drop, merge and skip updates for anyone who falls behind, and the score is still right.

What's being probed: whether you spot the "latest value wins" property and use it to simplify delivery.

Concept — state stream versus event stream. Some data is a sequence where every item matters: chat messages, payments, commentary lines. Lose one and the reader is wrong. Other data is a current value that keeps being replaced: a score, a stock price, a delivery driver's location. For a current value, a reader who missed three updates only needs the newest one. R18's stock-broking walkthrough uses the same idea for market prices.

Takeaway: Classify each stream as "every item matters" or "latest wins" before choosing how to deliver it.

Capacity

Interviewer: Size it.

You: Assumptions: one match with 50M concurrent viewers at the peak; a T20 innings has 120 legal deliveries plus extras in roughly ninety minutes, so an update arrives about every 30–40 seconds once commentary and other changes are included — call it one every 30 seconds. A compressed snapshot of the live view is about 2 KB; a delta (only what changed) is about 150 bytes. A handful of matches live at once.

The write side, so we can stop worrying about it:

T20 match: 2 innings x ~125 deliveries (incl. extras)   = ~250 ball events
+ commentary, reviews, breaks, corrections              = ~400 events per match
~1 KB per event                                          = ~400 KB per match
Ten matches live at once, one event per 30 s each        = ~0.3 events/s
History: assume 100,000 matches x ~1 MB (Tests are longer) = ~100 GB

Option A — every client polls a cached snapshot through a CDN:

50,000,000 viewers polling every 5 s           = 10,000,000 requests/s at the CDN
the score changes every ~30 s, so 5 of 6 polls see no change
  changed poll:   full 2 KB snapshot
  unchanged poll: 304 Not Modified, ~0.3 KB of headers
  average = (1/6 x 2 KB) + (5/6 x 0.3 KB)       = ~0.58 KB per poll
10,000,000/s x 0.58 KB                          = ~5.8 GB/s = ~46 Gbps
origin load: snapshot TTL 1 s, ~10 shield caches = ~10 requests/s at our origin
staleness: up to 1 s (TTL) + up to 5 s (poll gap) = worst ~6 s, average ~3 s

Option B — push over persistent connections (WebSocket or Server-Sent Events) through a fan-out tree:

connections per edge server (assumption, tuned)   = 250,000
edge servers = 50,000,000 / 250,000               = 200, plus 50% headroom = 300
memory: ~20 KB per connection x 250,000           = ~5 GB per server
per ball: 50,000,000 deltas x 150 B               = 7.5 GB
delivered within ~1 s                             = ~60 Gbps burst, ~300 Mbps per edge server
average: 7.5 GB every 30 s                        = ~250 MB/s = ~2 Gbps
messages: 50M per 30 s + heartbeats 50M per 30 s  = ~3.3M small frames/s
tree: 1 hub -> 20 relays -> 200 edges -> 250,000 viewers each
latency: a few hops of tens of milliseconds        = well under 2 s

Therefore:

• Writes are trivial. All the engineering is on the read side.
• Polling is simple and rides on a CDN we already have, but it costs about twenty times the bandwidth of push and is several seconds slower. Most of those polls ask "anything new?" and hear "no".
• Push is cheaper per update and faster, but it is a stateful fleet of 300 servers holding 50M connections, and reconnect storms become our worst failure.
• So use both: push for the app's live match screen, the cached snapshot for first load, web widgets, partners, and as the fallback when push fails.

What's being probed: whether you can compare two delivery models with numbers instead of opinions.

Concept — paying for "nothing changed". Polling costs grow with the number of viewers times the poll rate, whether or not anything happened. Push costs grow with the number of viewers times the number of real updates. When updates are rare compared with the poll rate — one every 30 seconds against a poll every 5 — push wins on bandwidth. The price is holding connections open, which is state, and state is what R1 and R18 warn is hard to scale and recover.

Takeaway: Count what each model costs per real update, then pay for state only where the saving is worth it.

API

Interviewer: Sketch the API.

You: Two very different audiences: a few scorers who write, and millions of viewers who read.

POST /v1/matches/m_wc26_final/events
Authorization: Bearer <scorer token>
{ "seq": 342, "epoch": 3, "type": "BALL",
  "innings": 2, "over": 17, "ballInOver": 4,
  "bowler": "p_301", "striker": "p_118", "nonStriker": "p_207",
  "runs": { "bat": 4, "extras": 0, "extraType": null }, "wicket": null }

HTTP/1.1 201 Created
{ "seq": 342, "status": "APPENDED" }

HTTP/1.1 409 Conflict
{ "error": "SEQ_GAP", "expectedSeq": 341 }

The sequence number doubles as the idempotency key: the scorer's tablet resends the same seq after a timeout, and the server answers "already have it" instead of recording a second four. A gap returns 409 with the sequence the server expects, so the tablet resends what is missing. A message on the viewer stream looks like this:

{ "m": "m_wc26_final", "seq": 342, "kind": "delta",
  "score": "172/3", "overs": "17.4", "crr": 9.74, "rrr": 11.14,
  "lastBalls": ["1", "W", "0", "4"], "striker": { "id": "p_118", "r": 41, "b": 27 } }

Data model

Interviewer: Data model.

You:

The ball log is the only truth; the scorecard, run rates, partnerships and career stats are all read models computed from it. If any of them is wrong, we fix the code and rebuild it from the log. Nobody ever edits a scorecard by hand.

High-level design

Interviewer: Draw it.

You: A narrow, strict write path into an ordered log, one builder per match turning events into state, and a wide read path with two lanes: a CDN for snapshots and a push tree for deltas. Play the flows: a ball end to end, a reconnect, a corrected ball, a wicket alert, then the final-over surge.

A ball, end to end:

1. The scorer taps "4". The tablet creates event seq 342 under writer epoch 3 and sends it.
2. The ingest API checks the epoch and that 342 follows 341, inserts the row and an outbox entry in one transaction, and answers 201. The outbox relay puts the event on Kafka, partition m_wc26_final.
3. The state builder for that match reads it, updates score, overs, batter and bowler figures, partnership and run rates, and gets state version 342.
4. It writes the new snapshot to the origin as latest and publishes a 150-byte delta to the hub.
5. The hub sends the delta to 20 relays; each relay to its ~10 edges; each edge to its ~250,000 connections. Push viewers see the four in about a second.
6. Polling viewers and widgets get the new snapshot on their next poll after the 1-second TTL expires.

What's being probed: whether the write path and read path are separated, and whether you can explain a tree instead of one giant broadcaster.

Concept — fan-out tree. No single machine can send 50M messages a second. So we send to a few relays, each of which sends to a few edges, each of which sends to its own viewers. Each level multiplies the reach by ten or twenty, and no node sends more than a few hundred thousand copies. It is how live-video CDNs distribute a stream too, and the same shape as Instagram's push fan-out with workers, except the "feed" here is a connection, not a stored list.

Takeaway: Fan out in levels so that no node's work grows with the whole audience.

Deep dive — one writer, one sequence

Interviewer: Why a single writer? Wouldn't two scorers be safer?

You: Two scorers typing at once would be two versions of the truth. If one records a dot ball and the other a single, which happened? Ordering is the game itself — a wicket on ball 3 versus ball 4 changes who faces next. So each match has exactly one active writer, and each event gets the next sequence number.

How we enforce it:

• A lease with an epoch. The scorer app takes a writer lease for the match. Each time the lease changes hands, the epoch goes up. Every event carries the epoch, and the ingest API rejects any event with an old one. This is a fencing token (R8): a tablet that lost its lease but still has events queued cannot overwrite the new scorer's work.
• A backup scorer. A second operator watches the broadcast and is ready to take over. If the primary's lease expires — its network died at the ground — the backup takes the lease, gets epoch 4, and continues from lastSeq + 1.
• Offline queueing on the tablet. Ground networks are patchy. The tablet numbers events locally and sends them in order; on reconnect it sends from the first unacknowledged one.
• One Kafka partition per match. Kafka keeps order only within a partition, so partitioning by matchId keeps every event of a match in order for every consumer.

The ingest check, in code:

@Transactional
public Ack append(String matchId, MatchEvent e) {
    MatchWriter w = writers.lockForUpdate(matchId);              // one row per match
    if (e.epoch() != w.epoch())       throw new StaleWriter(w.epoch());        // fenced off
    if (e.seq() <= w.lastSeq())       return events.sameAs(matchId, e)         // a retry?
                                          ? Ack.duplicate(e.seq())
                                          : reject("SEQ_REUSED");
    if (e.seq() != w.lastSeq() + 1)   throw new SeqGap(w.lastSeq() + 1);       // resend from here
    events.insert(matchId, e);                                   // PK (match_id, seq)
    outbox.insert(matchId, e);                                   // relayed to Kafka, key = matchId
    w.setLastSeq(e.seq());
    return Ack.appended(e.seq());
}

The duplicate branch checks the stored event matches the resent one; the same sequence with different content is a bug or a fenced writer, and is refused loudly rather than quietly accepted.

What's being probed: ordering and single-writer discipline, and how to fail over without two writers.

Concept — single writer per key. Concurrency bugs disappear when only one actor may write a given thing. The hard part is handing over when that actor dies: use a lease so the old owner's rights expire, and an epoch so anything it sends late is rejected. R8 covers leases and fencing tokens; R18's WhatsApp and Google Docs walkthroughs use one owner per conversation or document in the same way.

Takeaway: One writer per match, a sequence number per event, and an epoch to fence off the writer you replaced.

Deep dive — the scorecard is a read model

Interviewer: Where does the scorecard come from?

You: It is never written directly. The state builder folds events into state: start from an empty match and apply each event in order. Score, overs, each batter's runs and balls faced, each bowler's overs, runs and wickets, the current partnership, fall of wickets, the current and required run rates — all derived. This is CQRS from R11: the write model is the log; the read model is whatever shape the screens need.

Some of the derived values, to show they are pure functions of the log:

overs bowled        = legal balls / 6, shown as overs.balls (104 legal balls -> 17.2)
current run rate    = runs / (legal balls / 6)
required run rate   = (target - runs) / (balls remaining / 6)
partnership         = runs and balls since the last wicket in this innings
bowler economy      = runs conceded / overs bowled

The builder checkpoints its state at the end of every over. If it crashes, a new builder loads the last checkpoint and replays at most a few events from the log. If we ship a bug — say wides were counted against the batter — we fix the code and replay the whole match, and every snapshot and stat comes out right.

Interviewer: The scorer entered a six, but it was a four. Or a wicket that the third umpire later overturns.

Never edit or delete a ball event in place — every consumer that already read it would silently disagree with the log.

You: The fix is a compensating event: a new event, with the next sequence number, that says "event 346 is void; here is what really happened".

{ "seq": 347, "epoch": 3, "type": "CORRECTION", "voids": 346,
  "replacement": { "type": "BALL", "innings": 2, "over": 18, "ballInOver": 1,
                   "runs": { "bat": 4, "extras": 0 }, "wicket": null } }

The builder handles it by going back to the checkpoint before 346 and refolding, skipping the voided event and applying the replacement. It produces state 347 with the score two runs lower, and a delta marked "corrected": true, so the app shows "Correction: last ball was 4, not 6" instead of a number that mysteriously drops. Viewers still never see sequence numbers go backwards — the correction is a newer version, not an older one.

Third-umpire reviews fit the same model: WICKET with status UNDER_REVIEW, then REVIEW_RESULT with upheld or overturned. An overturned wicket is just another compensating event.

The log keeps the full history — what was entered, when, by whom, and what replaced it — which is also our audit trail when a fan writes in to argue.

What's being probed: event sourcing, and whether corrections are designed in rather than patched on.

Concept — compensating events. In an append-only log you never rewrite the past; you add a new fact that cancels an old one. Accountants work this way — a wrong ledger entry is reversed by an opposite entry, not rubbed out. It keeps every reader consistent, because they all see the same sequence, and it keeps the audit trail. R5's sagas use compensation for the same reason; R11 covers read models rebuilt from events.

Takeaway: Correct by appending, rebuild state by refolding, and tell the viewer it was a correction.

Deep dive — getting it to fifty million screens

Interviewer: Walk me through delivery in detail. Poll or push?

You: Both, each where it is strongest.

The snapshot lane (pull). Every state version is written to object storage as latest, served through the CDN with a 1-second TTL and an ETag equal to the sequence. The CDN collapses requests: however many viewers ask, each edge cache fetches from the origin at most once per second, through a shield layer, so the origin sees tens of requests a second. Clients send If-None-Match and usually get a tiny 304. This lane needs no state of ours at all, which is why it is the first load for everyone, the only lane for web widgets and partners, and the fallback whenever push fails.

The push lane. The app's live screen opens one WebSocket (or SSE, which is plain HTTP and passes more corporate proxies, and is enough because data only flows to the viewer). It subscribes to the matches on screen. The edge keeps a map of matchId → connections. Each edge subscribes upstream only to matches it has viewers for, and so does each relay, so a minor match with ten thousand viewers never touches most of the fleet.

Conflation — latest value wins. Suppose a viewer on a weak 3G connection cannot keep up, and three deltas queue for them. We do not send all three. The edge keeps, per connection, only the newest pending state for each match; when the socket can accept data, it sends that. Three updates become one. The same happens one level up: if a relay falls behind, it merges deltas before forwarding.

Conflation turns a slow reader from a growing queue into a slightly stale screen. That is the payoff of "latest wins": a slow client costs us nothing extra, instead of filling memory until the server falls over.

Commentary is the exception. Every line matters, so it is not conflated. The push lane sends the newest line; if the client sees a gap in commentary sequence numbers, it fetches the missing over from the CDN, where completed overs are cached for hours.

Resume after a drop. The client remembers the last sequence it applied. On reconnect — tunnel, lift, switching from Wi-Fi to mobile — it sends from=339. Each edge keeps a ring buffer of the last ~100 deltas per match:

• If 340 onwards is still in the buffer, the edge sends just those deltas.
• If the gap is older than the buffer, or the client's sequence is unknown, the edge replies "resync" and the client fetches the snapshot from the CDN, then continues on the stream.
• The client ignores any delta whose sequence is not newer than what it shows, so a delta and a snapshot arriving in either order never move the score backwards.

Two reconnects, side by side:

client shows 339 --reconnect from=339--> edge buffer has 290..342
edge sends 340, 341, 342                 client shows 342

client shows 120 --reconnect from=120--> edge buffer has 243..342
edge says RESYNC; client GETs snapshot (seq 342) from CDN; stream continues at 343

What's being probed: delivery mechanics at extreme fan-out — conflation, resume and the fallback lane.

Concept — conflation and resume-from-sequence. Conflation means merging queued updates into the latest one, which is only safe for "latest wins" data. Resume means a client says "I have up to 339" and gets only what it missed, or a fresh snapshot when the gap is too big. Together they mean disconnections and slow networks cost little. Kafka consumers resume from an offset in the same way (R9).

Takeaway: Number every version, buffer a few recent deltas for resume, and conflate everything that is "latest wins".

Deep dive — the hot match

Interviewer: India need 12 off the last over. Everyone opens the app at once.

You: Three things are true about cricket traffic that shape the answer. The schedule is known weeks ahead. The audience ramps sharply at the toss, at the start of each innings and in a close finish. And when something fails, the clients come back all at once.

So:

1. Pre-scale, do not autoscale. Autoscaling reacts in minutes; a close finish ramps in seconds, and a new connection server is useless until clients reconnect to it. We keep a capacity calendar: for a final, edge and relay fleets are provisioned for the forecast peak plus headroom an hour before the toss, CDN capacity is reserved, and the snapshot object is already warm in every cache.
2. Isolate the hot match. Give it its own relays and edge pool — a bulkhead (R14) — so its surge cannot starve the other live matches, and a bug triggered by its load stays inside its pool.
3. Tame reconnect storms. If an edge server with 250,000 viewers dies, all of them reconnect within seconds. Clients use exponential backoff with random jitter (R14), the load balancer spreads them across the remaining edges, and each edge accepts new connections at a capped rate. Clients waiting to reconnect fall back to polling the CDN snapshot, so they still see the score.
4. Shed to the cheap lane. When the edge fleet passes, say, 85% of its connection budget, new viewers are told to poll instead of connecting. The CDN can absorb them; our stateful fleet cannot grow on demand.
5. Degrade gracefully. Under pressure, raise the snapshot TTL from 1 s to 3 s, slow commentary pushes to batches, and turn off non-essential animations and side panels. The score is the last thing to degrade.
6. Freeze change. No deploys to the delivery path during big matches. A deploy drains connections, and draining 50M connections is itself a reconnect storm.

What's being probed: planning for predictable peaks and for the synchronised failure of a stateful fleet.

Concept — thundering herd on reconnect. When many clients lose a connection at the same moment, they all retry at the same moment, and the retries can knock over the servers that survived. Random jitter spreads them out; a cheap fallback gives them something useful while they wait. R12 covers the same herd effect on caches, and R14 covers backoff and admission control.

Takeaway: Pre-provision for known peaks, isolate the hot match, and give every client a stateless fallback while it waits.

Deep dive — commentary, clips, stats and alerts

Interviewer: What else hangs off the ball log?

You: Several consumers, each reading the same Kafka partition at its own pace — the point of putting a durable log in the middle (R9).

• Commentary. Commentators write a line per ball in a desk tool, attached to the ball's sequence, and it is published as a COMMENTARY event. Stored by (matchId, seq). Completed overs are immutable pages on the CDN with long TTLs; a correction to an old over publishes a new version of that page with a new URL, so no cache has to be purged. Other languages are separate commentary streams on the same balls; the score itself is language-neutral data that the app renders.
• Clips. The video team cuts a clip of the wicket; a CLIP_AVAILABLE event links it to the ball. Storage and delivery are R20's job — object storage, transcoding, CDN. Rights rules decide which regions see it.
• Stats. The stats pipeline consumes events into a columnar warehouse, one row per ball, for queries such as "most sixes in the death overs this season". Career figures are precomputed per player into a key-value store. During a match the app shows "career runs" as the pre-match total plus the live innings, so nothing heavy is recomputed per ball. Corrections flow through the same pipeline, because they are events too. After the match, the official scorecard is reconciled against ours, and any difference becomes a correction event (R6's idea, applied to sports data).
• Alerts. The state builder recognises key events — a wicket, a fifty or hundred, the end of an innings, the result — and publishes them to a key-events topic with a dedup key such as m_wc26_final:342:WICKET. The notification system subscribes. It finds who follows this team or match, checks their preferences, and sends a push to people who are not in the app. If the wicket is later overturned, it sends a follow-up using the same collapse key, so the phone replaces the first alert instead of stacking two.

The live view reaches people who are already watching; a wicket alert has to reach people who are not. That is the notification system's job, and the next walkthrough.

Failures and scale

Interviewer: What breaks?

You:

Curveballs

Interviewer: Our app is ahead of the TV stream, and users complain that we spoil the wicket.

You: Broadcast and streaming video run behind real time, by different amounts. Give the user a "sync with my stream" setting: the app holds each update for a chosen delay before showing it. It is all client-side, because the stream already carries timestamps; the server stays the same.

Interviewer: A betting partner wants the feed faster than anyone else.

You: That is a different product with a different contract: a dedicated authenticated feed straight off the hub, with its own capacity and SLA, priced accordingly. It must not share the consumer fleet, and the policy question — whether to offer it at all — is the business's, not the architecture's.

Interviewer: Could we just use a managed real-time pub/sub service?

You: For the push lane, quite possibly — the fan-out tree is what such services run internally. The design keeps the parts that are ours: single writer, sequence numbers, the read models, conflation semantics and the CDN fallback. Check its limits on connections per channel, fan-out rate and cost at 50M before trusting it with a final.

Interviewer: Two thousand small matches at once, each with a few hundred viewers.

You: Now the problem is subscriptions, not bandwidth. Most edges have viewers for some matches; relays subscribe per match on demand, so a match nobody on an edge watches never reaches it. State builders are cheap — one consumer group spreads the partitions over a few dozen instances.

Quick reference

The 60-second version

The scorer's tablet is the single writer for a match: it holds a lease with an epoch, numbers every event, and the ingest API accepts only the next sequence number, so resends are harmless and a replaced scorer is fenced off. Events go into an append-only log, one Kafka partition per match. A state builder per match folds them into the live view and scorecard — run rates, partnerships, figures — checkpoints every over, and can rebuild from the log at any time; a wrong ball is fixed by a compensating event, never an edit. Each new state is written as a snapshot behind a CDN with a one-second TTL, and pushed as a small delta through a tree — hub, relays, edge servers holding 50M WebSocket or SSE connections. At 50M viewers polling would cost about 46 Gbps and several seconds; push costs about 2 Gbps and one second, but is stateful, so the CDN stays as first load and fallback. Because every update supersedes the last, slow clients are conflated to the latest state and reconnecting clients resume from their last sequence or resync from the snapshot. Known peaks are pre-scaled, the hot match gets its own pool, reconnects are jittered, and overload sheds viewers to polling. Commentary, clips and stats hang off the same log, and key events such as a wicket go to the notification system to reach people who are not watching.

Walkthrough 3 — Notification system: push, SMS and email at scale

Building on live scores: live scores pushed the same tiny update to everyone who was already watching, over connections we held open ourselves. A notification system has to reach people who are not in the app. That means going through providers we do not control — Apple's and Google's push services, SMS aggregators, email services — each with its own limits, failures and costs. And every user is different: their own channels, preferences, language and quiet hours. The fan-out is no longer one value copied to everyone; it is millions of individually decided messages, each of which must arrive once, on time, and only if the user wants it.

Scope and requirements

Interviewer: Design a notification system for a large consumer app — say, a payments and shopping app in India.

You: The core is turning an event — "₹1,250 debited", "your order shipped", "Diwali sale starts now" — into the right message on the right channel for each user, sent exactly once as far as the user can tell. A few questions:

Interviewer: Internal services and a campaign tool for marketing. Mobile push, SMS, email and an in-app inbox. OTPs, payment alerts, order updates and promotions. 200 million registered users; broadcasts up to 50 million users. You own preferences and consent. India first, other countries later. A duplicate payment alert is bad; a missed one is worse; a duplicate promotion is just annoying.

You: Requirements.

Functional:

• Accept a notification request: a user, a category, a template and its parameters.
• Decide eligibility: the user's preferences and opt-outs, quiet hours, frequency caps and legal consent.
• Render the template per channel and language.
• Deliver through mobile push (APNs for iOS, FCM for Android), SMS, email and the in-app inbox, with fallback between channels where the category allows.
• Run campaigns to a segment of users, scheduled and throttled, with a pause button.
• Track status per message: accepted, sent, delivered, opened, failed — and why.
• Let users manage preferences and see past notifications in an inbox.

Non-functional:

• Never send the same notification twice because of our own retries; and never lose a transactional one.
• OTP delivered within about 10 seconds at p95; payment alerts within 30 seconds; campaigns within the window the marketer sets.
• A 50M-user broadcast must not slow down a single OTP.
• Survive any one provider failing.
• Respect every opt-out, quiet hour and regulatory rule — a mistake here is a legal problem, not just a bug.
• Cost-aware: SMS costs real money per message.

The hard part is not sending a message; it is deciding, for each user and each attempt, whether to send it at all.

What's being probed: whether you see the system as policy plus delivery, not just "a queue in front of an SMS API".

Concept — categories drive everything. A notification's category — OTP, transaction alert, order update, promotion — decides its priority, which channels it may use, whether the user can opt out, whether quiet hours apply, how long it stays useful, and whether to risk a duplicate rather than a miss. Get the category model right and most later decisions become table lookups.

Takeaway: Ask what kinds of message exist and how each fails badly, before drawing any boxes.

Capacity

Interviewer: Size it.

You: Assumptions: 200M registered users, 60M daily active, about 1.5 push-enabled devices per active user. Transactional traffic: 100M notifications a day — 50M payment alerts, 30M order updates, 20M OTPs — with an evening peak five times the average. Promotions: about 200M a day across campaigns, the biggest a 50M-user broadcast that marketing wants out within 30 minutes. About 1.3 delivery attempts per notification (several devices, some fallbacks). For sizing only, assume an SMS costs 15 paise.

transactional: 100,000,000 / 86,400 s        = ~1,160/s average
peak (x5)                                     = ~6,000/s
OTP: 20,000,000 / 86,400 s                    = ~230/s average, ~1,200/s peak

broadcast: 50,000,000 users / 1,800 s         = ~28,000 users/s
x 1.5 devices                                 = ~42,000 push sends/s to APNs and FCM

notifications a day: 100M transactional + 200M promotional = 300M
attempts: 300M x 1.3                                       = ~390M a day
record per attempt incl. status events: ~1 KB             = ~390 GB a day
90 days hot                                                = ~35 TB before replication

OTP SMS: 20,000,000 a day x 15 paise   = 300,000,000 paise = Rs 30,00,000 (30 lakh) a day
a bug that duplicates 1% of them        = Rs 30,000 a day wasted, 200,000 confused users

broadcast side effect: assume 3% of 50M open the app within 5 minutes
1,500,000 opens / 300 s                 = ~5,000 extra app opens/s
x ~10 API calls per open                = ~50,000 extra requests/s on our own backend

Therefore:

• Our own servers are not the bottleneck. Provider limits, money and our own app backend are.
• A broadcast is five times the peak transactional rate. If they share a queue, OTPs wait behind "Diwali sale". So: separate lanes.
• Duplicates have a price in rupees and in trust, so dedup is a first-class feature, not a nice-to-have.
• Broadcasts must be throttled for our backend's sake as well as the providers'.

What's being probed: whether you find the real bottleneck instead of sizing servers.

Concept — the external bottleneck. When a system's job is to call other systems, its throughput is set by theirs: an SMS aggregator's contracted messages per second, a push service's quotas, an email provider's sending reputation. And a notification causes load elsewhere — every push that is tapped opens the app. Size the things you do not control first. R17's rate limiter is what keeps us inside those limits.

Takeaway: Size providers, costs and side effects first; the queue and workers are the easy part.

API

Interviewer: API.

You:

POST /v1/notifications
Idempotency-Key: payments:txn_8812731:DEBIT_ALERT
{ "userId": "u_4410", "category": "TRANSACTION_ALERT",
  "template": "debit_alert",
  "params": { "amount": 125000, "currency": "INR", "merchant": "Sharma General Store",
              "account": "XX4021" },
  "expiresAt": "2026-10-09T14:10:00Z" }

HTTP/1.1 202 Accepted
{ "notificationId": "n_01JA3K9", "status": "ACCEPTED" }

-- the payments service times out and retries with the same key
HTTP/1.1 200 OK
{ "notificationId": "n_01JA3K9", "status": "SENT", "duplicate": true }

The idempotency key comes from the business event — the transaction id plus the kind of alert — not from a random id the caller generates per attempt. A fresh UUID on every retry would defeat the whole mechanism. The amount travels in paise, and the template formats it as ₹1,250.00 in the user's language. expiresAt says when the message stops being useful; a payment alert delivered tomorrow is noise.

Data model

Interviewer: Data model.

You:

The idempotency key is the line between "we may retry" and "the user sees it once". Everything before it is free to retry; everything after it checks the key or the attempt's status before acting.

High-level design

Interviewer: Draw it.

You: An API that accepts and deduplicates, a planner that decides who gets what on which channel, Kafka lanes by priority, one worker pool per channel talking to providers through rate limiters and circuit breakers, and a tracker that turns provider callbacks into status. Play the flows: a payment alert, a retried request, an OTP when the SMS provider fails, a 50M broadcast, and dead tokens.

A payment alert, end to end:

1. The payments service commits a debit and, through its outbox, calls POST /v1/notifications with key payments:txn_8812731:DEBIT_ALERT.
2. The API sets the key in Redis only if absent; it is new, so it writes the notification row (unique on the key) and returns 202.
3. The planner loads the category TRANSACTION_ALERT: high priority, push plus SMS, users cannot opt out, quiet hours do not apply. It loads the user's language (Hindi) and two device tokens, renders the push and SMS variants, and writes one attempt per channel and device.
4. Attempts go onto the transactional lane, keyed by user id.
5. The push worker checks the provider rate limiter and sends to APNs and FCM. The SMS worker sends through aggregator A with our registered sender id and template.
6. Providers accept; the tracker records SENT with each provider message id. Aggregator A's delivery receipt arrives a few seconds later: DELIVERED. The user opens the app: OPENED. A copy sits in the inbox.

What's being probed: a clean pipeline in which policy, queuing and delivery are separate stages.

Concept — decide once, deliver many times. Eligibility and rendering happen once, in the planner, and produce fully formed attempts. Workers then only deliver: they do not look up preferences or templates. That keeps workers simple and fast, makes every attempt self-contained for retries, and means a preference change during a retry storm does not cause inconsistent decisions. It is the same split as R9's producer and consumer: put a durable log between deciding and doing.

Takeaway: Make every policy decision before the queue, and let the workers only deliver.

Deep dive — lanes: an OTP must never wait behind a sale

Interviewer: Why not one queue?

You: Because a 50M broadcast enqueued at 7 pm would put 50M messages ahead of every OTP for the next half hour. We split by category into lanes:

Separate lanes means separate Kafka topics, separate worker pools and separate provider capacity — a bulkhead (R14). It also means separate sending identities. For SMS, transactional and promotional messages travel under different registered sender ids and routes. For email, marketing is sent from a different subdomain and IP pool from receipts and alerts, so that if a campaign gets marked as spam, the payment receipts still reach the inbox.

Within a lane, messages are keyed by user id so one user's messages stay in order — "shipped" before "delivered". Across users, there is no ordering to protect.

An OTP has one more rule: it expires. A worker that picks up an OTP after its expiresAt drops it and records EXPIRED, because a code that arrives after the login screen timed out only confuses the user.

Never let promotional traffic share workers, topics or sender identities with OTPs and payment alerts.

Deep dive — sending once

Interviewer: The payments service retries. A worker crashes halfway through. How do you not send twice?

You: Duplicates can enter at three places, and each needs its own guard.

1. The caller retries. The API does SET key notificationId NX EX 86400 in Redis. If the key already exists, it returns the existing notification id and status, and does nothing else. Redis is the fast path; the unique constraint on (caller, idempotencyKey) in the notification table is the backstop if Redis loses the key. This is R3's idempotency key.

2. Kafka redelivers. Kafka gives at-least-once delivery (R9): a worker that crashes before committing its offset will see the same attempt again. So each attempt has a status, and a worker claims it with a conditional update before calling the provider:

UPDATE attempt SET status = 'SENDING', worker = :me, claimed_at = now()
 WHERE notification_id = :n AND channel = :c AND attempt_no = :a
   AND status = 'PENDING';
-- 1 row: we own it, send. 0 rows: someone already did, skip.

3. The provider call has an unknown outcome. The worker sent the SMS, and then crashed or timed out before recording SENT. Did it go? We cannot know (R0's unknown outcome). Most push and SMS APIs cannot be asked "did you already receive this?", so the choice is a policy per category:

• OTP: a duplicate is better than a miss. After a short wait, retry — through the other provider if the first looks unhealthy. Make the duplicate harmless: resends within the validity window carry the same code, so two SMS show the same six digits.
• Payment alert: check for a delivery receipt for the stored provider message id first; if none arrives within a minute, retry once.
• Promotion: a miss is better than a duplicate. Mark it UNKNOWN and do not retry.

For push, we also set a collapse identifier — APNs and FCM both support one — equal to our notification id. If the same push arrives twice, the phone shows one notification, not two. The in-app inbox is keyed by notification id, so it can never hold duplicates.

Campaigns use the key campaignId:userId, so re-running a crashed batch never messages anyone twice.

There is no exactly-once delivery to a system you do not control; there is dedup on our side, harmless duplicates where they slip through, and a per-category choice between a duplicate and a miss.

What's being probed: idempotency end to end, and honesty about the unknown outcome at the provider boundary.

Concept — effectively once. A message system can deliver at-least-once (retry until acknowledged) or at-most-once (never retry). Exactly-once is achieved only by making repeats harmless: dedup keys, conditional state changes and idempotent receivers. Where the receiver is a phone company's network, we cannot make it idempotent, so we make the content harmless instead — the same OTP, a collapse id. R3 covers idempotency keys; R0 covers the unknown outcome; Datadog's alert notifications, next, face the same problem.

Takeaway: Guard each place a duplicate can enter, and decide per category whether to risk a duplicate or a miss.

Deep dive — preferences, quiet hours, templates and the rules

Interviewer: How does the planner decide what to send?

You: In a fixed order, cheapest checks first, and it records the reason for every message it does not send:

1. Category rules. Is this channel allowed for this category? Has the message expired?
2. Consent and opt-outs. Promotions need the user's opt-in per channel. Transactional and security messages cannot be switched off, but the user can choose push instead of SMS where the category allows. Every consent change is stored with a timestamp and source, because we may have to prove consent later.
3. Quiet hours. In the user's timezone — not the server's. A promotion at 11 pm goes to the scheduler and is released at 9 am, unless it expires first. OTPs and fraud alerts ignore quiet hours.
4. Frequency caps. At most, say, three promotions per user per day across all teams — a per-user counter in Redis, the rate limiter from R17 applied to a person. Without it, five teams each sending "just one" campaign add up to a user who uninstalls.
5. Reachability. Does the user have a live push token, a verified phone number, a non-bounced email? If not, fall back to the next allowed channel.

Templates and localisation. Templates are versioned and approved before use; a request names a template and parameters, never free text, so a caller cannot send arbitrary content. Each template has variants per channel — a push has a title and body, an SMS is short plain text, an email has a subject, HTML and plain text — and per language, with a fallback chain such as hi-IN → hi → en. Plurals and number formats use a message-format library, not string concatenation, and amounts are formatted from paise at render time.

SMS length is a cost question. A single SMS holds 160 characters in the basic GSM alphabet but only 70 in Unicode, which Hindi and other Indian scripts need; longer messages are split into several billed parts. So a Hindi OTP template is kept short on purpose.

Regulation, described generally. Rules vary by country and change, so they live in configuration and in legal review, not in code. The shapes to design for:

• Sender and template registration. In India, commercial SMS runs under the telecom regulator's rules on a distributed-ledger (DLT) registry: the business, its sender ids and each message template must be registered with the operators, and messages that do not match a registered template are blocked. So our template store keeps the registration id per SMS template, and a template cannot go live until its registration is approved. WhatsApp business messages have a similar pre-approved template model.
• Do-not-disturb and time windows. Promotional messages respect national do-not-disturb preferences and allowed sending hours; transactional messages have different rules.
• Email. Authenticate the sending domain (SPF, DKIM, DMARC), include a working one-click unsubscribe on marketing mail as large mailbox providers now require, and suppress addresses that bounce or complain.
• Content. Never put sensitive data in a notification — full account numbers, balances on a lock screen — because lock screens and SMS inboxes are not private.

What's being probed: whether user trust and legal rules are part of the design, not an afterthought.

Concept — policy as data. Preferences, categories, quiet hours, caps and regulatory rules change far more often than code. Keep them as data the planner evaluates, with an audit trail, so a new rule is a configuration change reviewed by the right people. Record why each message was not sent; when a user or a regulator asks "why did I get this?" — or "why didn't I?" — you can answer.

Takeaway: Evaluate policy in one place, from data, and log the reason for every decision.

Deep dive — providers: limits, failover and callbacks

Interviewer: SMS aggregator A starts timing out during the evening peak.

You: Each provider sits behind an adapter with four protections.

• Rate limiting. Each SMS aggregator sells a contracted throughput; push services enforce quotas; email providers limit by account and reputation. A token bucket per provider and route (R17) keeps us under those limits, and honours 429 and Retry-After when they push back.
• Retries by error class. Transient errors — timeouts, 5xx, 429 — retry with exponential backoff and jitter (R14), up to a limit. Permanent errors — invalid number, unregistered token, template rejected — never retry; they update the token or contact record. After the last retry the attempt goes to a dead-letter queue with its error, for people to inspect and replay. An expired OTP is never replayed.
• Circuit breakers. Per provider (R14). When aggregator A's error rate or latency crosses a threshold, the breaker opens and the SMS workers route to aggregator B; after a cool-down a few probe messages test A again. Routing is weighted in normal times too, by cost and by each aggregator's recent delivery rate per mobile operator.
• Callbacks. Providers report final outcomes later through webhooks — SMS delivery receipts, email bounces and complaints. The callback endpoint verifies the provider's signature or credentials, writes the event to Kafka and answers 200 at once. The tracker looks up our attempt by (provider, providerMessageId) and applies the update idempotently.

Callbacks arrive late, twice, or out of order — DELIVERED before SENT is common. So status only moves forward:

stateDiagram-v2
    [*] --> PENDING
    PENDING --> SENDING: worker claims
    SENDING --> SENT: provider accepted
    SENDING --> RETRY_WAIT: transient error
    RETRY_WAIT --> SENDING: backoff elapsed
    RETRY_WAIT --> DEAD_LETTER: retries exhausted
    SENDING --> FAILED: permanent error
    SENT --> DELIVERED: receipt
    SENT --> FAILED: receipt says failed
    DELIVERED --> OPENED: user opened
    PENDING --> EXPIRED: past expiresAt

A provider that accepts messages and quietly fails to deliver them is worse than one that returns errors. Its API looks healthy, so an error-rate breaker never trips. The breaker must also watch the delivery-receipt rate: if aggregator A's receipts fall from 97% to 60% in ten minutes, shift traffic to B.

Push needs one more nuance. APNs and FCM tell us they accepted a message, not reliably that the user saw it, and they keep only limited copies for an offline device — APNs keeps just the most recent one per app. For the few categories where receipt matters, the app acknowledges the notification with a small API call, and the in-app inbox is the durable copy.

What's being probed: resilience against dependencies you do not control, and knowing that "accepted" is not "delivered".

Concept — measure the outcome, not the call. A successful API call only means the provider took the message. The real outcome — delivered, bounced, opened — arrives later through callbacks or the app. Health checks, circuit breakers and SLOs should be based on outcomes. R14 covers breakers and backoff; R15 covers SLOs that measure what users experience.

Takeaway: Fail over on delivery rate as well as errors, and treat every callback as late, duplicated and out of order.

Deep dive — the 50-million broadcast and device tokens

Interviewer: Marketing wants to tell 50M users about the Diwali sale at 7 pm.

You: A broadcast is a batch job with brakes.

1. Materialise the segment. The campaign's audience is a query — "active in 90 days, opted in to promotional push, city in these states". A job runs it against the warehouse and writes user ids in chunks of 10,000 to object storage. The audience is fixed at that moment, so a retry sends to the same people.
2. Canary first. Send to 1% and watch the funnel for ten minutes: delivery rate, open rate, uninstalls, complaints, and our own backend's load. A wrong discount code reaches half a million people, not fifty million.
3. Release in throttled batches. The scheduler hands out chunks at the campaign's rate, under a global bulk budget shared by all campaigns, as jobs with leases (R17's job scheduler), so a crashed worker's chunk is picked up again. Each user goes through the planner — opt-in, caps, quiet hours, timezone — so the eligible count is lower than the segment.
4. Protect our own backend. Our earlier sum showed a broadcast can add tens of thousands of requests a second as people tap through. The send rate is chosen with that in mind, the landing pages are pre-warmed in the CDN, and the app team knows it is coming.
5. Keep the kill switch live. The scheduler checks the campaign's status before releasing each chunk, and workers check before each send. Pausing stops the rest within seconds. What was already sent cannot be recalled — push and SMS have no "unsend" — so a correction goes only to the users the status store says received the original.

Throughput is bounded by providers, not by us. Push services can take high volumes, but each has quotas; SMS broadcasts at this size are rarely worth their cost, so campaigns default to push, email and the inbox.

Device tokens are what make push work, and they rot:

• The app registers its token on every launch, because both platforms can change a device's token. POST /v1/devices upserts by token, which is unique: if the same phone logs in as a different user, the token moves to the new user.
• Logout unbinds the token. Otherwise the next person to use that phone — or the buyer of a second-hand one — receives the previous owner's payment alerts. For a payments app, that is a privacy breach, not a glitch.
• When APNs answers 410 Unregistered, or FCM reports a token as unregistered, the tracker marks the token dead at once. Tokens not refreshed for, say, 60 days are pruned. Sending to dead tokens wastes quota and can count against us with the provider.
• The same discipline applies to the other channels: a hard email bounce suppresses the address; repeated SMS failures to a number flag it for re-verification.

What's being probed: the operational side of fan-out — throttling, safety brakes, and keeping addresses healthy.

Concept — throttle for the whole system. A broadcast's speed is limited by the slowest thing it touches: provider quotas, money, and the surge of users it brings back to the app. Canary, ramp and a kill switch make a mass send as safe as a code deploy (R16 uses the same ideas for releases). It is Instagram's celebrity problem in reverse: there, one post had to reach everyone quickly; here, we deliberately slow one message down so the rest of the system survives it.

Takeaway: Freeze the audience, canary, release in throttled batches, keep a kill switch, and keep tokens clean.

Failures and scale

Interviewer: What breaks?

You:

Curveballs

Interviewer: A user says they got the same OTP three times.

You: Likely an unknown-outcome retry plus a failover to the second aggregator — the policy for OTPs, by design. Because resends within the validity window carry the same code, it is harmless. If it happens often, the funnel will show which provider is slow to accept, and we tune its timeout.

Interviewer: Add WhatsApp as a channel.

You: A new channel adapter and worker pool, a channel column in preferences with its own opt-in, and template variants that go through WhatsApp's own template approval. The planner, lanes, dedup and tracker do not change — that is the test of the design.

Interviewer: The bank needs a debit alert for every transaction, by regulation.

You: Then that category is non-optional and must reach the user on a channel that does not depend on the app being installed — typically SMS, with push as an extra. We keep evidence: the attempt, the provider's acceptance and the delivery receipt, retained as long as the compliance team requires.

Interviewer: Why not just let each team call FCM and the SMS provider directly?

You: Then every team re-implements dedup, preferences, quiet hours, caps, registration rules, failover and token clean-up — and the user gets five teams' promotions in an hour with no cap across them. One platform enforces the user's wishes and the law in one place.

Quick reference

The 60-second version

Callers raise notifications from business events with an idempotency key derived from the event, and the API deduplicates it in Redis with a unique constraint as the backstop. A planner applies everything that is policy — category rules, consent, quiet hours in the user's timezone, frequency caps, reachability — renders versioned, approved templates per channel and language, and writes self-contained attempts onto Kafka lanes: critical for OTPs, transactional for alerts, bulk for campaigns, each with its own workers and sender identities so a 50M broadcast never delays a login. Workers claim each attempt with a conditional update and send through provider adapters with token-bucket limits, backoff with jitter, a dead-letter queue and circuit breakers that watch delivery rate as well as errors, failing over between two SMS aggregators. Because the provider boundary has unknown outcomes, each category chooses between a duplicate and a miss, and duplicates are made harmless with the same OTP code and push collapse ids. Callbacks are verified, processed idempotently and only move status forward, feeding a requested-to-opened funnel. Broadcasts freeze their audience, canary at 1%, release throttled chunks with a live kill switch, and are paced to protect our own backend. Device tokens are refreshed on launch, unbound on logout and removed when the provider says they are dead.

Walkthrough 4 — Datadog, monitoring and observability at scale

Building on Instagram, live scores and notifications: the mirror image. Instead of one write reaching many readers, millions of hosts write into one platform, and every choice is about making the write cheap — aggregation at the source, a buffer that takes the acknowledgement, compression, rollups — and about the skewed tail, which here is high-cardinality tags rather than celebrities. Datadog is also R15 seen from the other side: the platform your SLOs and alerts run on.

R15 is how you use observability: the three signals, golden signals, SLOs, alerts on symptoms. This walkthrough is building the platform those things run on, which is why several of its traps — averaging percentiles, alert noise, the monitor that cannot see its own outage — are R15's traps seen from the inside.

Scope

Interviewer: Design Datadog.

You: "Observability" is several products with different storage engines, so I'll scope it. Which signals? — metrics, logs and traces each need a different store. Multi-tenant SaaS? — keys, quotas, noisy neighbours, per-tenant billing. Resolution, retention, freshness? — tiers, rollups and the latency budget. Our own agent? — it can pre-aggregate at the edge. How do we bill? — the main lever on tag cardinality. Alert semantics? — streaming, periodic, or both.

Interviewer: Multi-tenant SaaS, all three signals, go where the hard parts are.

You: ~30K customer organisations, from 5 hosts to 100K+. A host agent collects everything, pre-aggregates and ships batches over HTTPS. Metrics at 10 s resolution, raw for 15 days, rollups for 15 months. Logs: customers choose what to index for 15 days; everything can be archived. Queryable and alertable within ~30 s of emission at p99. Traces are accepted and sampled; their deep design is out of scope. I'll go deep on metrics, then logs.

Interviewer: Why an agent? Why not API push, or scrape like Prometheus?

You: Raw push from apps turns every StatsD call — a busy app emits 100K a second — into traffic to us. Pull is excellent inside one company network, where the server controls the rate and knows when a target dies, but a SaaS cannot reach targets behind customer firewalls and NAT. A local agent receives app metrics over UDP or a Unix socket, pre-aggregates, batches, compresses and pushes outbound over HTTPS, which crosses firewalls, and buffers to disk when the network fails. It cuts intake load by roughly a thousand times.

What's being probed: narrowing a huge product, and seeing that the collection model sets the whole load profile.

Concept — one front door, several engines; push from the edge. Metrics are small numbers over time read as ranges; logs are semi-structured text searched by keyword; traces are trees of spans linked by an id — three stores sharing one ingest path. For collection, pull suits a single network and push suits a SaaS the receiver cannot reach; an edge agent gives apps something cheap and nearby and sends one compact batch across the internet.

Takeaway: For SaaS telemetry, an edge agent pushing compressed batches is the default; name which engine you will go deep on.

Requirements

Interviewer: What must it do, and what matters most?

You: Agents and SDKs send gauges, counts, rates and distributions, logs and traces, all tagged: host:web-1, env:prod, service:checkout. Users query any metric over any range, filtered and grouped by tags, with aggregations and functions; build auto-refreshing dashboards; create monitors — threshold, change, anomaly, no-data — that notify Slack, PagerDuty, email, webhooks. Logs get parsing pipelines, selective indexing, search, live tail, archive and rehydrate. Per-tenant metering and quotas. Out of scope: APM internals, RUM, synthetics, security products, invoicing.

Tags make metrics flexible to query and expensive to store: every unique combination of name and tags is a separate stored series, and that trade-off drives the rest.

Non-functionally: ingest availability 99.99%; p99 under 30 s emit-to-queryable and alert evaluation under 60 s; dashboard widgets p95 under 1 s over an hour, 5 s over 30 days; acked data replicated before the ack; one tenant's spike must not slow others; ~1–2 bytes per stored point; write-only API keys, application keys and SSO for reads. The priority order is: don't lose data, then freshness for alerts, then query speed.

What's being probed: capturing the read side, and ranking qualities instead of "everything fast".

Concept — asymmetric availability. A dashboard slow for a minute makes people wait; telemetry dropped at intake is gone forever, possibly with the outage it would have shown. So the write path gets the strictest availability and durability and the read path may degrade — partial results, stale cache. This is R15's error-budget thinking applied per path.

Takeaway: Protect the write path first; the read path can bend.

Capacity

Interviewer: Numbers.

You: Assumptions: 5M agents, 500 active series each after agent aggregation, 10 s flush, ~10 KB compressed per flush, ~2 bytes per stored point, 2x peak; logs 5M events/s at 1 KB with 20% indexed and 10:1 archive compression; 10M monitors every 60 s; 100K dashboard queries/s at peak.

Bytes are cheap; distinct series are expensive. Index size grows with the number of series, not points, so a customer sending a few points each for millions of new series costs far more than one sending many points for a few. The 500 GB of active index matters more than the petabytes, because it must live in memory.

API

Interviewer: The main APIs.

You: Batch writes — POST /api/v2/series, /api/v2/logs, /api/v0.2/traces with a DD-API-KEY — and reads: POST /api/v1/query for metrics, /api/v2/logs/events/search with facets and a cursor, CRUD on /api/v1/monitor and /api/v1/dashboard.

POST /api/v2/series
DD-API-KEY: 4f1c...
Content-Encoding: zstd

{ "series": [
  { "metric": "system.cpu.user", "type": "gauge", "interval": 10,
    "points": [{"timestamp": 1790000000, "value": 37.5}],
    "tags": ["host:web-1", "env:prod", "service:checkout"] },
  { "metric": "checkout.requests", "type": "count",
    "points": [{"timestamp": 1790000000, "value": 1240}],
    "tags": ["env:prod", "endpoint:/pay", "status:200"] } ] }

HTTP/1.1 202 Accepted

A query reads like avg:system.cpu.user{env:prod,service:checkout} by {availability-zone} over a range and interval. The API key is baked into thousands of agents and will leak, so it can only write — a mail slot. Reads use an application key or SSO with RBAC.

Interviewer: Why 202? What about retries and floods?

You: 202 means "durably in Kafka, not yet queryable" — intake stays fast and decoupled from storage, and the agent only needs to know it can drop its local buffer. Gauges keyed by (series, timestamp) are idempotent on retry; counts are not — adding 1,240 twice gives 2,480 — so the agent attaches a batch id and intake dedups over a short window (R3). Over quota, 429 with Retry-After; the agent buffers on disk and retries with jittered backoff (R14). Payloads over ~5 MB compressed get 413. Log search uses a cursor, never offset.

Data model

Interviewer: Entities and stores.

You:

series_key = org_id + "|" + metric_name + "|" + sort(tags).join(",")
series_id  = 64-bit hash(series_key)        // plus a collision check in the index

system.cpu.user{host:web-1,env:prod} and {host:web-2,env:prod} are two series. A query is two steps with two structures: find series with an inverted index, read their points from a TSDB.

High-level design

Interviewer: End to end.

You: Split the system at a durable buffer: a thin, highly available intake on the left, specialised consumers for metrics, logs and alerts on the right. Play the write path and then the alert path, and watch where the 202 is returned — before any storage is touched.

The write path in eight steps:

1. App calls statsd.increment("checkout.requests", tags=["endpoint:/pay"]); the agent aggregates over 10 s — 1,240 increments become one count point — and attaches host tags.
2. At flush it sends ~500 series as zstd-compressed protobuf with its key; on failure, disk buffer and backoff.
3. GeoDNS picks the region; intake checks the key from cache and the org's token bucket, appends to Kafka, returns 202 after acks=all.
4. A writer finds or creates series_id (bumping cardinality counters) and appends to the replicated head block. Queryable within seconds; the streaming evaluator on the same topic may update alert state at once.
5. Later, compaction seals 2 h blocks, computes 1-min and 1-h rollups, and tiers old data.

What's being probed: decoupling ingest from processing, and why the buffer exists.

Concept — the log as the spine. Kafka between intake and storage gives four things at once: durability once it acks, spike absorption when writers lag, replay after a bad deploy, and independent fan-out to writers, log pipeline and evaluator. R9 covers the mechanics; the design point is where the ack sits.

Takeaway: Ack after the durable log, not after storage; everything downstream can then fail and recover.

Deep dive — agent pre-aggregation and sketches

Interviewer: A busy app emits 100K StatsD calls a second. How do you not drown?

You: Aggregate at the agent. Counts summed per window, gauges last value, distributions into sketches — DDSketch, publicly described by Datadog, gives a relative-error guarantee and merges. One compressed payload per host per 10 s. Intake per request is small: key lookup from a cache refreshed every ~30 s, a local token bucket synced to a global counter, size check, append to Kafka with producer batching. If Kafka is down, intake returns 503 and the agent keeps buffering — durability lives in two places, the agent's disk and Kafka. The key maps to org_id, which travels with every message; nothing downstream trusts tags for tenancy.

Never average per-host p99s

A host with 10 requests would count the same as one with 10 million, and percentiles do not average anyway. This is R15's latency trap at the storage layer: ship a mergeable sketch, merge thousands of them in the query engine, and get one correct global p99.

What's being probed: edge aggregation, mergeable summaries, and a thin hot path.

Concept — mergeable sketches. A sketch summarises a distribution, for example counts in logarithmic buckets, such that combining two sketches gives the sketch of the combined data. DDSketch places bucket boundaries so any reported percentile is within a fixed relative error, say 1%.

Takeaway: Ship summaries that merge, never pre-computed percentiles.

Deep dive — Gorilla compression, tiers and rollups

Interviewer: You claim ~2 bytes a point. A timestamp and a float64 are 16.

You: Gorilla compression, published by Facebook in 2015. Timestamps as delta-of-delta: points every 10 s give deltas 10, 10, 10, so delta-of-delta is 0, one bit. Values XORed with the previous one: slowly changing floats share sign, exponent and most mantissa bits, so the XOR is mostly zeros and only the meaningful bits are stored. The paper reports ~1.37 bytes a point; I assume 2 with overhead.

encode_timestamp(t):
  d = t - prev_t; dod = d - prev_d
  if dod == 0:                write '0'
  elif -63  <= dod <= 64:     write '10'   + 7 bits
  elif -255 <= dod <= 256:    write '110'  + 9 bits
  else:                       write '1110' + 12 bits   (or '1111' + 32 bits)

encode_value(v):
  x = bits(v) XOR bits(prev_v)
  if x == 0:                                   write '0'
  elif leading/trailing zeros fit prev window: write '10' + meaningful bits
  else:                                        write '11' + 5 bits leading + 6 bits length + meaningful bits

The index keeps a posting list per tag value — env:prod → {s1, s2, s5}, service:checkout → {s2, s5, s9} — as compressed Roaring bitmaps, so env:prod AND service:checkout is a fast intersection, {s2, s5}.

Rollups store sum, count, min and max, not the average, so any aggregation is correct at the coarser level: avg = sum ÷ count, max of maxes; distributions merge sketches. The planner picks the tier from the range and the pixels: a 30-day graph about 300 points wide needs ~2.4 h per point, so it reads 1-h rollups and never raw data.

What's being probed: time-series compression and how tiers keep cost flat.

Concept — compress the predictable part. Steady intervals make the change in the gap almost always zero; slow-moving values make neighbouring XORs mostly zero. Tiering applies the same idea over time: old data is rarely read in full detail, so keep summaries — and keep them mergeable, because averages of averages are wrong.

Takeaway: Encode deltas, roll up with sum/count/min/max, read the coarsest tier the screen can show.

Deep dive — tag cardinality

Interviewer: A customer adds user_id as a tag on a request metric.

You: One metric goes from 50 series to 50 million. Each costs index memory, a head block and write amplification, and the tenant's queries and shard neighbours suffer. Cardinality is the product of tag value counts: endpoint 20 × status 5 × host 500 is 50,000; add a million user_ids and it is 50 billion. Control it in layers:

on new series(org, metric, tags):
  hll[org][metric].add(series_id)
  if new_series_rate[org] > org_limit or hll[org][metric].count() > metric_limit:
     emit_usage_event(org, metric, "cardinality_limit")
     if policy == DROP_NEW:  return reject
     if policy == AGGREGATE: tags = drop_non_allowed(tags)
  index.create(series_id, tags)

Interviewer: The customer says per-user data is critical and they'll churn if you drop it.

You: Then I redirect it rather than drop it silently. Per-user questions belong in logs and traces, which are built for high-cardinality ids and billed by volume, not series. A log-based metric keeps user_id in the log and aggregates the metric by a bounded tag like plan_tier. If they truly need it as a metric, it is a paid, explicitly configured exception, sharded wide, with its own limits so it cannot hurt neighbours.

What's being probed: the most important failure mode of a metrics system, and product judgement under pressure.

Concept — cardinality. Each series needs an index entry and a write buffer whether it gets one point or a million, so cost tracks series count, not data size. HyperLogLog estimates distinct counts within a few percent in kilobytes, which makes per-metric tracking cheap. The fix is layered: price it, limit it, aggregate it, and steer unbounded ids elsewhere.

Takeaway: Never put unbounded ids in metric tags; that is what logs and traces are for.

Deep dive — query engine and dashboard caching

Interviewer: Execute sum:checkout.requests{env:prod} by {endpoint}.as_rate() over four hours.

You:

1. Parse and normalise; pick the rollup interval: 4 h ÷ 240 points = 60 s.
2. Index lookup returns matching series ids grouped by endpoint.
3. Fan out to the TSDB shards that own them. Each shard downsamples and pre-aggregates by group, returning (group, interval) → (sum, count) — a map-side combine that turns millions of points into a few thousand partials.
4. The coordinator merges: sums and counts add, max of maxes, sketches merge.
5. Apply as_rate(), formulas between queries, top-N.

Datadog's query language publicly separates time aggregation (within a series per interval) from space aggregation (across series). Protection: estimate cost as series × points before running and force a coarser rollup above a limit; per-tenant concurrency limits and timeouts; hedged requests to replicas for tail latency; partial results with a warning if a shard is slow (R14's scatter-gather).

Interviewer: 20 widgets, thousands of viewers, auto-refresh every 30 s.

You: A whole-result cache never hits, because "last 4 hours" at 10:00:00 and 10:00:30 are different ranges. But the past does not change. Cut the range into wall-clock-aligned buckets and every bucket except the newest is identical between the two requests. Key = hash(org, normalised query, rollup interval, bucket start); closed buckets cached in Redis with LRU; the open bucket always recomputed; buckets newer than a lateness window (say five minutes) never cached as final, and a late write invalidates its buckets. Popular NOC dashboards hit almost 100%, and the browser reuses its previous points so only the delta is fetched. Materialised dashboards would hit more but only for known queries, and waste work. R12 covers cache keys and invalidation in general.

What's being probed: scatter-gather with partial aggregation, the avg-of-avgs trap, and caching a moving key.

Concept — mergeable partials and aligned buckets. Push as much aggregation as possible to the shards, but only in forms that merge: the average of one item at 10 and 99 items at 0 is not (10 + 0) ÷ 2, so return sum and count and divide at the end. For caching, fixed wall-clock buckets turn a key that moves every request into a stream of immutable chunks plus one live one.

Takeaway: Return mergeable partials from shards; align time to fixed buckets so the past becomes cacheable.

Deep dive — alert evaluation

Interviewer: 10 million monitors. How do you evaluate them?

You:

Hybrid. Simple single-metric thresholds with short windows stream, with state keyed by monitor and group and checkpointed. Everything else runs periodically, sharded by monitor_id across workers holding leases, so each monitor has exactly one evaluator at a time (R8). Evaluate the window [now − W − d, now − d], where d (say 60 s) covers ingestion lag, so a half-filled last minute does not drag the value down or fake a no-data.

The alert path:

1. avg(last_5m):avg:system.cpu.user{env:prod} by {host} > 90, warn at 80, no-data after 10 minutes; stored in Postgres, assigned by hash(monitor_id).
2. Every 60 s the evaluator queries [now − 5m − d, now − d], one value per host.
3. Per group it compares to thresholds and computes OK, WARN, ALERT or NO DATA against the previous state in Redis.
4. host:web-7 goes OK → ALERT: a transition, so an event. Groups staying in ALERT send nothing unless renotify is configured.
5. The notifier builds a dedup key, drops duplicates, groups hosts alerting in the same minute, applies per-tenant rate limits.
6. It calls Slack and PagerDuty with retries and idempotency keys, recording the notification.
7. ALERT → OK sends a recovery, which auto-resolves the PagerDuty incident.

What's being probed: latency versus flexibility, and keeping alerts stable.

Concept — evaluation delay and transition-only notification. Data arrives slightly late, so evaluate slightly in the past. Each monitor group is a small state machine, and notifying only on transitions turns 167K checks a second into a trickle of meaningful events — R15's rule that an alert must mean a human should act, enforced by the platform.

Takeaway: Evaluate slightly in the past, and notify on state changes, not on every check.

Deep dive — no-data, our own lag, and effectively-once notification

Interviewer: Your Kafka consumers fall ten minutes behind during an incident. What happens to every customer's no-data monitor?

You: Naively, every no-data monitor across every tenant fires at once and we page thousands of on-call engineers for our own outage.

A monitoring platform's own lag looks exactly like every customer's hosts dying

So: track last_seen per (monitor, group). Before declaring no-data, check the platform's own ingestion lag for that org or partition; if it is high, suppress or mark evaluations as delayed rather than paging. Groups that vanish because autoscaling removed hosts auto-resolve after a timeout. When data catches up, evaluation resumes from the right window and replays never re-trigger historical alerts.

On the output side, exactly-once delivery to Slack is impossible; aim for effectively once. Dedup key = (monitor_id, group, state, transition_time), stored for a window and reused on retry; PagerDuty also accepts a dedup key. Recovery thresholds and "N consecutive bad evaluations" stop flapping; grouping collapses many groups into one message; downtimes mute planned work.

What's being probed: knowing that the platform's failures can masquerade as customer failures.

Concept — absence of evidence and effectively-once. "No points from this host" could mean the host died or the points are stuck in our pipeline; the data alone cannot tell, so the evaluator needs a second signal, our own lag. On the output side, effectively once is at-least-once delivery plus a stable key the receiver dedups on — R3's idempotency applied to a page.

Takeaway: Check your own lag before blaming the customer's host; exactly-once is at-least-once plus a dedup key.

Deep dive — logs, noisy neighbours and late data

Interviewer: Logs are 5 GB/s. Indexing all of it is very expensive.

You: Decouple ingestion from indexing — Datadog's publicly described "Logging without Limits". Ingest and process everything; let the customer decide what is worth searching.

ingest -> parse (grok/JSON) -> enrich (host tags) -> log-based metrics
       -> exclusion filters / sampling -> index (15 d, searchable)
       -> archive (everything, object storage)       -> live tail (stream, no storage)

Parsing is a stateless pool scaled by Kafka lag, with customer pipelines compiled and cached. Log-based metrics — say, count of status:error by service — keep 15-month trends for logs never indexed. The index is time-partitioned with columnar attributes for facets and a token index for full text; expired partitions drop whole, no per-document deletes. Archives are compressed per org per hour; rehydration reads only the hours in range into a temporary index. Sensitive-data scanning masks PII before indexing. A one-hour search touches only a few partitions and streams newest-first with a cursor.

Interviewer: One tenant is 100x the median. And agents that were offline replay hours of data.

You: Isolation at every layer — R14's bulkheads:

Late data: the head accepts out-of-order points within a window of an hour or two, but Gorilla assumes order, so they go into a side "late" chunk merged at compaction. Older points take a late-write path that rewrites the sealed chunk and recomputes rollups asynchronously, invalidating cache buckets. Points too old (over a day) or too far in the future (over 10 minutes of clock skew) are rejected or clamped and counted. Monitors never retroactively fire for a historical window.

What's being probed: cost-aware log design, multi-tenant isolation, and correctness when time is not monotonic.

Concept — keep versus make searchable; cells; a side door for late data. Indexing is the expensive part of logs; a compressed archive is cheap, so treat them as separate decisions. A cell is a complete stack for a subset of tenants, so a bad deploy or runaway tenant sinks only its compartment. Append-optimised encodings assume order, so late points get a side path merged later rather than breaking the fast path.

Takeaway: Ingest everything, index what's worth searching; limit blast radius with cells; give late data a side door.

Scale and failure

Interviewer: What breaks first?

You:

TSDB sharding: hash(org_id, metric) to a shard group, then hash(series_id) within the group for huge metrics, with consistent hashing and virtual nodes for rebalancing (R10). Sealed chunks in object storage make moving shards cheap.

Interviewer: Region failure, data residency, and who monitors the monitor?

You: Each org is homed in one region or site — Datadog publicly runs separate sites such as US1, US3, US5, EU1, AP1 — and its data stays there. Within a region, Kafka, TSDB and index replicas are spread across three AZs, and multiple cells limit blast radius. RPO for telemetry is seconds, because the agent buffer and Kafka hold unprocessed data; RTO for losing a whole region is hours, while agents keep buffering. Object storage and Postgres backups replicate cross-region. And the monitoring system needs an independent watcher — a separate small stack watching ingestion lag and evaluation health — so an outage cannot hide itself. It is R15's "measure from where the customer is", applied to the monitor itself.

Curveballs

Interviewer: Why build a custom TSDB? Why not Cassandra or Postgres?

You: You can — OpenTSDB used HBase. But generic stores pay per-row overhead, cannot use delta-of-delta and XOR encoding, and have no inverted tag index, so storage is 5–20x bigger and "all series with env:prod" becomes a scan. At 250M points a second that difference is a hardware budget. Without a custom engine I'd use Cassandra or Bigtable wide rows and accept 5–10x the storage, or ClickHouse at moderate scale.

Interviewer: If you had a small team, what would you cut?

You: Every component here exists because a number forced it, so cut the ones whose number is not there yet: purely periodic evaluation instead of hybrid; ClickHouse instead of a custom TSDB; and direct writes to storage instead of Kafka only if the system is genuinely small, accepting no replay and fragility under spikes. What I would not cut, at any size: write-only keys, per-tenant quotas, and monitors in a separate query pool from ad-hoc exploration. Knowing which boxes exist because of scale proves you know the reasons, not just the boxes.

Quick reference

The 60-second version

An agent on every host pre-aggregates metrics into 10 s windows, with sketches for distributions, batches and compresses them, and pushes to a regional intake, buffering on disk if the network fails. Stateless intake checks a write-only key and per-tenant quota, appends to Kafka and returns 202 once it is durable there. Writers map metric plus tags to a series id through an inverted index and append into a custom TSDB with Gorilla compression at about two bytes a point, tiered from memory to SSD to object storage, with sum/count/min/max rollups for 15 months. Cardinality is the cost driver, controlled by billing, limits, HyperLogLog and ingest-time aggregation. Queries fan out to shards that return mergeable partials, and dashboards cache closed time buckets. Monitors evaluate slightly in the past, notify only on transitions with dedup keys, and suppress no-data when our own lag is high. Logs are ingested in full, indexed selectively and archived cheaply. Cells and quotas contain noisy neighbours, and an independent stack watches the watcher.

What to say in the interview

"High-volume flow is about shape, so I name it first: fan-out, where one write must reach many readers, or fan-in, where many writers pour into one store. Then I decide where the work happens — on write or on read — and use a hybrid for the skewed tail: push for the many small authors, pull for the few huge ones; aggregate at the source for most metrics, and limit the few tags that explode cardinality. A durable log sits in the middle and takes the acknowledgement, so bursts are absorbed and consumers can replay. I store ids, aggregates and mergeable sketches rather than full copies, and choose freshness per use. Live scores push the latest state through an edge tree and let late joiners start from a snapshot; notifications decide eligibility once, deduplicate on a key and keep OTPs in their own lane, failing over between providers. In Datadog the cost is distinct series rather than bytes, aggregates have to be mergeable, and the platform checks its own lag before it pages anyone."