Design a Product Recommendation System

Who Asks This Question?

This shows up at any company whose revenue depends on what a user sees next: e-commerce (Amazon product recommendations, "customers also bought"), video (YouTube, Netflix, TikTok "up next"), social feeds (Meta, Instagram), and music (Spotify Discover Weekly). It is also asked in several disguises that are all the same system:

• Design Amazon's product recommendations
• Design the YouTube up-next feed
• Design a personalized news feed ranker
• Design Spotify's Discover Weekly

The core architecture is identical: a retrieval-then-ranking funnel over a large catalog, fed by an event and feature pipeline. Only the deep-dive focus changes.

This question tests whether you understand the gap between "computing similar items" (easy) and "ranking a billion items for hundreds of millions of users in under 200 milliseconds while behavior changes by the second" (hard). Companies ask it to see if you have hit real production challenges: serving latency, feature freshness, training-serving skew, and feedback loops. Master the general version and you answer all the variants.

What the Interviewer Is Really Testing

The reliability row is where most candidates lose points. Anyone can name a ranking model; few can keep the system fast, fresh, and correct at once while the model behind it changes weekly.

Step 1: Clarify Requirements

Functional Requirement

Ask a sharp question, commit to an answer, move on. Scope first, then objective, then the core action:

• Which surface are we building? The homepage feed, the "customers also bought" strip, search re-ranking, and email are really different systems, so I want to pin this down before I draw anything. Let me assume the homepage feed, and I'll call out where item-to-item would change things.
• What's the business objective? I'd optimize for conversion, not clicks. Clicks are easy to get but you end up with clickbait people tap and leave; conversion is what actually makes money. The tradeoff is that purchase signal is rarer and slower to come back than clicks, so it's harder to learn from.
• So what does the system hand back? For a given user and page, a ranked list of N items, ordered by how much we think each one moves that goal. Under the hood we predict a few things per item (will they click, buy, stay, come back) and combine them into a single score to sort by.
• Should it react to what someone just did? Yes. If you clicked on something a couple seconds ago, the next page should reflect that. That's what forces us to keep features fresh in near real time, not just refresh once a day.
• Shall we show items that are out of stock, already bought, or blocked in the user's region? No, we filter those out before the list goes back.
• What about cold start, brand-new users and items? They have no history to learn from, so for them we lean on where they are and what's generally popular until we've seen enough of their behavior to personalize.

Non-Functional Requirements

Each follows from a functional requirement above, ordered from the constraints that force the architecture down to the ones that just shape it:

• Scale (from serving every user on every page): 100M daily active users, a catalog up to 1B items, roughly 150K requests per second at peak (about 30 recommendation-bearing page loads per user per day). This is what makes the problem hard at all; at small scale everything is trivial.
• Latency (from returning a list on every request): p99 under roughly 200 milliseconds online. At that scale, this budget is what forces the multi-stage funnel: you can't score a billion items in 200ms.
• Freshness (from real-time session signals): recent behavior reflected within seconds, not only on the next day's batch refresh. This forces the two-clock feature pipeline, streaming plus batch.
• Availability (from never showing a blank module): degrade gracefully; a recommendation outage should downgrade quality, not remove the feature. This shapes the degradation tiers rather than the core skeleton.
• Measurability (from optimizing conversion): every change proven by an A/B test. This needs serving support: a router that splits traffic by variant, and impression logs tagged with the variant so outcomes can be compared.

Step 2: High-Level Architecture

Start with the happy path: a request reaches the recommendation service, which retrieves candidates, ranks them, filters, and returns the top N. Get that skeleton on the board, then refine it against the requirements: the latency budget forces a funnel, and freshness forces a streaming feature path.

The organizing idea to say out loud: the client talks to two entry services, and everything else hangs off them. The Recommendation Service answers "what should I see" in milliseconds; the Event Service ingests "what I did" into the data plane. Models are assumed pre-trained and available; how they are produced is out of scope.

Components

The components group into two services plus a shared model boundary. Recommendation Serving owns the online request path (orchestrator, cache, retrieval, ranking); the Event Platform owns event ingestion, the streaming and batch feature jobs, and the feature store. Models are assumed pre-trained and published to a Model Registry that serving loads at deploy time. The two services communicate through the feature definition and the event schema.

Recommendation Serving (the online request path, scaled by QPS):

• API Gateway + Load Balancer: single client entry; routes to the two entry services (Recommendation and Event).
• Recommendation Service: stateless online orchestrator. Checks the cache, runs the funnel, applies filters, logs impressions. It orchestrates inference; it does not run models itself.
• Retrieval Service (candidate generation): two-tower retrieval over the Vector Index (FAISS / ScaNN); turns a 1B-item catalog into a few thousand plausible items. The index is sharded, since roughly 500 GB at 1B items does not fit one machine.
• Ranking Service: horizontally scaled ranking-model pods behind an inference gateway, with a fallback pod for degradation and an A/B router that splits traffic across model versions.
• Cache (Redis): precomputed recommendation lists and hot embeddings; the fast path.

Event Platform (the data plane, scaled by event volume):

• Event Service: ingests client behavior events (click, add-to-cart, purchase) into the event stream.
• Event Stream (Kafka / Kinesis): carries every interaction and impression; the entry to the data plane.
• Stream Processor (Lambda): consumes Kafka / Kinesis, computes real-time session features into the hot store within seconds.
• Batch Processor (Glue / Spark): computes long-term features from the raw event logs on a schedule.
• Feature Stores: items store (batch-updated), cold user store (batch, long-term features), hot user store (Redis, streaming session features). Owns the one feature definition the serving path reads.
• Object Store (S3): raw event logs, the durable source of truth.

Model Registry (boundary): holds the pre-trained model versions; the Retrieval and Ranking services load them at deploy time. It is populated by an offline training pipeline that is out of scope here.

Step 3: Deep Dive on Core Component Design

1. The Online Serving Path

The path serves each request synchronously within 200ms. Since a billion items cannot be scored in that time, it is a funnel: each stage narrows the set cheaply enough that the next, more expensive stage can afford it. It runs as three units, each with its own model and feature stores so they scale and deploy independently: a thin stateless Recommendation Service as the front controller (cache, filtering, re-ranking, impression logging), a Retrieval Service for candidate generation, and a tier of GPU Ranking pods behind an Inference Gateway for scoring.

Walking one request through the path:

1. Cache check. The user hits the Recommendation Service, which checks the Recommendation Cache (Redis) and returns immediately on a hit.
2. Retrieval. On a miss, the Recommendation Service calls the Retrieval Service, which reads the user's features (session features from the Hot User Feature Store in Redis, long-term features from the User Feature Store in DynamoDB), runs the two-tower user tower to get a query embedding, and queries the Vector Index by approximate nearest neighbor for the closest item embeddings; those item embeddings are precomputed offline, so only the one user vector is computed live. It merges in cached popularity and co-purchase lists (co-purchase is item-to-item collaborative filtering) and trims to about 100 with a lightweight gradient-boosted tree. Two-tower over plain collaborative filtering because it takes content and context features, so it handles new users and items and reacts to the live session; item-to-item CF survives as just one merged source.
3. Cold start. When the user is brand-new (no session or history) or an item was just added, its two-tower embedding is weak or absent, so retrieval leans on context and popularity candidates (region, current trends, onboarding signals) and reserves an exploration slot to gather the first interactions; personalization takes over once enough signal accrues.
4. Ranking. The Retrieval Service forwards the candidates to the Inference Gateway, which fans out across N Ranking pods (with a Fallback pod for degradation). Each pod reads item features (Hot Item Feature Store in Redis, Item Feature Store in DynamoDB) and runs the transformer ranker, returning per-item predictions (click, purchase, dwell, return) rather than one number. Both retrieval and ranking load their model weights from the Model Registry at deploy time, not per request.
5. Filter and re-rank. The scored candidates return to the Recommendation Service, which filters against PostgreSQL (drop already-purchased via purchase history, drop out-of-stock via inventory), combines the predictions via the value model, applies diversity and business rules, and returns the top N. (Hard filters can be pushed earlier into retrieval to avoid ranking ineligible items; here they sit at re-rank for simplicity.)
6. Cache and log. It writes the result to the Recommendation Cache and logs an impression to the event stream (Kafka / Kinesis), asynchronously and off the response path.

The API surface:

POST /v1/recommendations
Authorization: Bearer <jwt>
Content-Type: application/json

{
  "user_id": "u_8f3a",
  "context": { "surface": "homepage", "device": "ios", "session_id": "s_11c9" },
  "count": 20
}

{
  "request_id": "req_4b2e",
  "items": [
    { "item_id": "p_9931", "score": 0.87 },
    { "item_id": "p_2210", "score": 0.81 }
  ],
  "model_version": "ranker_2026_07_05"
}

The impression event, logged after the response:

{
  "event": "impression",
  "request_id": "req_4b2e",
  "user_id": "u_8f3a",
  "items": ["p_9931", "p_2210"],
  "positions": [1, 2],
  "model_version": "ranker_2026_07_05",
  "ts": "2026-07-06T14:22:31Z"
}

2. Event and Feature Pipeline (Two Clocks)

Everything the serving path reads is precomputed here. The Event Service ingests user interactions, the Recommendation Service logs impressions, and both land on the same event stream (Kafka / Kinesis), which then splits into two clocks: a streaming path for freshness and a batch path for durability and heavy features.

• Fast clock (seconds). A Lambda consumer reads the stream and writes real-time features into Redis, the Hot User Feature Store (recent views, current-session category) and the Hot Item Feature Store (live counts), so a click is reflected in the next request without waiting for the daily batch. (A stateful processor like Flink is the alternative when features need large windowed state.)
• Slow clock (daily). Kinesis Firehose buffers the same stream into S3 raw storage, the durable source of truth. On a schedule, EventBridge triggers a Glue / Spark job that reads S3, computes long-term features (7-day view counts, category affinity), and writes the cold User Feature Store and Item Feature Store in DynamoDB.

The stores split by update cadence, not read speed: hot stores are Redis (streaming writes, sub-millisecond reads on the serving path), cold stores are DynamoDB (batch writes, larger and cheaper for the full user and item base). All carry the same feature definition, illustrated here as SQL:

CREATE TABLE item_features (
  item_id      BIGINT PRIMARY KEY,
  category_id  INT,
  price_cents  INT,
  avg_rating   REAL,
  views_7d     BIGINT,
  in_stock     BOOLEAN,
  embedding    VECTOR(128),
  updated_at   TIMESTAMPTZ
);

Step 4: Optimization & Trade-offs (Where You Win the Interview)

Walk the system against the non-functional requirements from Step 1 and name the trade-off on every choice.

Latency

The p99 budget is roughly 200ms end to end, and it is spent, not abundant. A rough split: ~10ms gateway and network, ~2ms cache lookup, ~15ms feature fetch, ~40ms retrieval (user-tower inference plus the scatter-gather ANN), ~60ms ranking (GPU scoring of ~100 items), ~10ms re-rank and filter, leaving headroom for a retry. Three levers hold the line:

• Parallelism. Fan out the hot, cold, and item feature reads concurrently and pay the slowest, not the sum; query the vector-index shards scatter-gather and merge. Nothing sequential that can be parallel.
• Tail control. p99 is set by the slowest shard and the slowest ranking pod, so the tail, not the mean, is the enemy. Hedge the ranking call (send to two pods, take the first to answer) and over-query shards (ask for one more than you need, drop the laggard).
• Timeouts into degradation. Every stage has a deadline; missing it falls back a tier (below) instead of blowing the whole budget.

Trade-off: hedging spends extra inference to cut the tail, and a tight timeout occasionally serves slightly worse ordering to protect the deadline.

Scalability

Size it from the numbers. A cache hit ratio of ~50% on (user, context, model-version) leaves ~75K QPS reaching the funnel; each request scores ~100 candidates, so the GPU fleet must sustain ~7.5M scorings per second. Dynamic batching at the inference gateway (coalescing requests into GPU-sized batches) is what makes that affordable, at the cost of a few ms of queueing, so cap the batch wait (this is the direct tension with the Latency section). The stateless orchestrator scales horizontally on QPS behind the load balancer.

The 500GB vector index is the other axis: shard it (say 8 shards of ~64GB) so each fits in memory, and replicate each shard (say 3x) both for read QPS and so a lost node degrades only one slice. Hot keys are the trap: a celebrity item or a viral query concentrates load on one shard, so cache the popular candidate lists and replicate hot shards harder.

Trade-off: batching buys throughput with a little latency; replication multiplies memory cost, and memory (the index) is the expensive resource here.

Freshness

Two clocks with explicit staleness bounds. Session features are fresh within seconds (bounded by stream-consumer lag); long-term features run up to ~24h stale (the batch cadence). That is fine for taste, but not for eligibility: stock and price change by the second, so the filter step reads them from an authoritative source (Postgres or a short-TTL cache), never from a possibly-stale feature store, or you recommend a sold-out item.

Trade-off: the streaming path is a second pipeline to operate, and the authoritative eligibility read costs latency; both are paid to avoid stale or wrong recommendations.

Fault Tolerance & Graceful Degradation

Degrade in tiers, each with a concrete trigger:

• Ranking slow or erroring (per-stage timeout, or a circuit breaker on error rate): serve lightweight-ranker order, or route to the fallback pod running a cheaper model.
• Retrieval down: serve precomputed category and popularity candidates.
• Everything down: serve a static cached bestseller list.

Under a 50x spike (Black Friday), add load shedding before the GPU fleet saturates: extend cache TTLs, raise the cache-served fraction, drop the exploration budget, and shed the funnel to its cheaper tiers. Stale or less-personalized is acceptable; a blank module never is.

Trade-off: every tier trades personalization for availability, and keeping a warm fallback pod costs idle capacity.

Consistency & Data Correctness

• Training-serving skew: one feature definition, and log the exact feature values used at serving time. Pin each model to a feature-schema version so the model and the features it expects roll out together, never half-and-half.
• Eligibility from source of truth: filter stock, price, and region against the authoritative store, not a stale mirror, so the funnel never surfaces something the user cannot buy.
• Duplicate and at-least-once: the stream delivers at least once and clients retry, so deduplicate impressions and interactions on request_id (an idempotency key) at consume time, so one retry is not counted as two events.

Cost

GPU inference dominates the bill, so the whole funnel doubles as a cost-control device: the cache absorbs ~half the traffic, the lightweight trim cuts the candidate set ~30x before any GPU touches it, dynamic batching keeps the GPUs saturated, and popularity and co-purchase lists are precomputed offline rather than per request. Autoscale the GPU fleet down off-peak. Trade-off: caching and precompute buy cost by spending freshness; the harder you cache, the staler the tail of results.

End-to-End Data Flow

A request enters through the gateway to the Recommendation Service, which checks the cache, fetches features, retrieves a few thousand candidates from the sharded vector index, trims them with the lightweight ranker, scores about 100 with the heavy ranker, re-ranks with the value model and filters, and returns the top N. The service logs the impression to Kafka / Kinesis; the client separately reports clicks and purchases through the Event Service into the same stream. Kafka / Kinesis persists raw events to S3 and feeds a Lambda consumer, which writes session features to the hot store in seconds. On the slow clock, Glue / Spark computes long-term features into the cold stores. The Retrieval and Ranking services load their pre-trained models from the Model Registry at deploy time; how those models are produced is out of scope.

Failure Scenarios

Common Mistakes

• Ranking the whole catalog. A billion items cannot be scored in 200 milliseconds; skipping the funnel misses the core insight.
• One service for requests and events. Ingestion spikes then degrade serving; separate the entry points.
• No impression logging. Without recording what was shown, you cannot A/B test model versions or measure whether recommendations worked.
• Leading with click-through rate. Rewards clickbait; progress to conversion with retention as a guardrail.
• Ignoring training-serving skew. Serving must feed the model the same feature values it was trained on; name the shared feature definition and serving-time feature logging.

Interviewer Follow-Up Questions

"How do you handle a 50x Black Friday spike?" Serve cached recommendations on a short time-to-live, shed the funnel to its cheaper tiers, and autoscale ranking pods. Personalization degrades; the module never goes blank.

"How do you recommend to a brand-new user?" Region and context popularity, onboarding signals mapped to user clusters, and an exploration budget to learn quickly.

"Why not compute features at request time?" Cost and skew: request-time computation blows the latency budget, and independently recomputed features drift from the definition the model was trained on. Precompute on two clocks and read from the store.

"How do you keep the two feature clocks consistent?" One feature definition shared by the streaming and batch paths, plus logging the exact serving-time values so the model sees the same features in production that it was trained on.

"How do you handle position bias?" The serving side logs the position of every impression, so the logged data records where each item was shown. Correcting for it (so the model learns preference, not layout) is the ML team's job; the systems requirement is simply that positions are captured in the impression log.

"How do you measure success?" Offline ranking-quality metrics gate a model before rollout (owned by the ML team). What the systems side owns is the online proof: the pipeline to log impressions, split traffic by model version, and run an A/B test on conversion and return rate, watching one short-term and one long-term metric.

Key Terminology

Summary: Your Interview Plan

Work the happy path first, then layer in reliability. And every time you draw a component, say "I chose X over Y because…"; proactively volunteering trade-offs is the strongest signal of principal-level thinking.