What is the retrieval, ranking, and re-ranking funnel in a recommendation system?

It is the multi-stage architecture every large recommender uses because you cannot score millions of items per request under a latency budget. Retrieval (also called candidate generation or sourcing) reaches into the millions and cheaply returns a few hundred to a few thousand plausible candidates using approximate methods like a nearest-neighbour vector index. Ranking then scores those candidates with a heavy model that reads hundreds of features — often a light first-stage ranker that trims to about a hundred, then a heavy second-stage ranker. Re-ranking takes the ordered list and reshapes it for diversity, freshness, and business rules before it reaches the screen. Instagram states the principle directly: "most large-scale recommender systems employ a multi-stage funnel approach — starting with thousands of candidates and narrowing down the number of candidates to hundreds as we go down the funnel." Each stage looks at fewer items than the last and spends more per item.

What is a two-tower model in recommendation systems?

A two-tower model is the dominant retrieval architecture. As Meta describes it, "The Two Tower model consists of two separate neural networks — one for the user and one for the item," trained so that similarity between the user embedding and the item embedding predicts engagement. The trick that makes it cheap enough for millions of items is separation: the item tower runs offline, so every item in the catalog is embedded once and loaded into an approximate-nearest-neighbour index (HNSW and relatives). At request time you only run the user tower — one forward pass — to get the user vector, then ask the index for its nearest item vectors. Scoring is a dot product, not a model call, so millions of items are effectively scored by geometry in a few milliseconds. It is the same retrieval machinery used in RAG systems.

What is a feedback loop (and position bias) in recommendation systems?

A feedback loop is the self-reinforcing bias that arises because the ranker is trained on logged behaviour — but users can only click items the current system chose to show them. The training data is a sample of the old policy's choices, not of the world. Two biases compound: presentation bias, where an item never shown can never be clicked so popular shown items accrue ever more evidence and the rich get richer; and position bias, where an item shown at the top gets clicked more because of its position, so the model mistakes position for quality. Left untreated the loop collapses the catalog toward a handful of hits and produces filter bubbles. The defences are to log the propensity (how likely each item was to be shown) so you can inverse-weight it, to deliberately explore uncertain items, and to log context honestly so the bias is measurable.

How do LLMs change recommendation systems?

They change what goes inside the funnel stages, not the funnel itself — nothing about a language model lets you score ten million items per request. Three shifts are real. Semantic retrieval uses LLM-quality embeddings so retrieval understands meaning rather than just co-occurrence; it slots into the existing two-tower-plus-ANN layer as one more source. LLM-as-ranker reads the user context and a candidate together like a cross-encoder and judges relevance far more sharply than a dot product, but it is far more expensive per item, so it lives at the narrow end of the funnel re-scoring the top handful, never the corpus. Generative recommendation, where a model emits an item's "semantic ID" directly the way it generates the next token, is genuinely new but still unsettled research, not the default. The hype to reject is "just ask an LLM to recommend from the catalog" — it cannot hold millions of items in context, cannot answer in the latency budget, and hallucinates items that do not exist.

What is train/serve skew in a recommendation ranker?

Train/serve skew is when the feature values a ranker was trained on differ from the values it is served in production, so a model that looked brilliant offline degrades live with nothing in the logs to point at. It happens when a feature like "user's clicks in the last hour" is computed one way by the batch job that builds the training set and a subtly different way by the service that computes it live at request time — a different time window, different rounding, or a value that had not settled yet. The model trained on one number and is served another; nothing errors, accuracy just quietly decays. The fix is to compute each feature once from a single definition and feed both training and serving from it, which is the core job of a feature store. It is one of the most common ways a recommender fails in production, which is why it is worth naming explicitly in an interview.

Design a Recommendation System (in the LLM Era)

Millions of items, one person, a hundred-millisecond budget — you can never score the whole catalog, so every large recommender is a funnel: cheap retrieval narrows millions to hundreds, a heavy ranker scores only the survivors, and a re-ranker shapes the final slate. We compute the scoring budget, dial an interactive funnel until it blows past 100ms, walk two-tower retrieval and the feature store behind the ranker, face the feedback loop that quietly collapses the catalog, and separate what the LLM era really changes from the hype — grounded in YouTube and Instagram's own papers.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design a Recommendation System (in the LLM Era) — built to be understood, not skimmed.

Questions

What is the retrieval, ranking, and re-ranking funnel in a recommendation system?: It is the multi-stage architecture every large recommender uses because you cannot score millions of items per request under a latency budget. Retrieval (also called candidate generation or sourcing) reaches into the millions and cheaply returns a few hundred to a few thousand plausible candidates using approximate methods like a nearest-neighbour vector index. Ranking then scores those candidates with a heavy model that reads hundreds of features — often a light first-stage ranker that trims to about a hundred, then a heavy second-stage ranker. Re-ranking takes the ordered list and reshapes it for diversity, freshness, and business rules before it reaches the screen. Instagram states the principle directly: "most large-scale recommender systems employ a multi-stage funnel approach — starting with thousands of candidates and narrowing down the number of candidates to hundreds as we go down the funnel." Each stage looks at fewer items than the last and spends more per item.
What is a two-tower model in recommendation systems?: A two-tower model is the dominant retrieval architecture. As Meta describes it, "The Two Tower model consists of two separate neural networks — one for the user and one for the item," trained so that similarity between the user embedding and the item embedding predicts engagement. The trick that makes it cheap enough for millions of items is separation: the item tower runs offline, so every item in the catalog is embedded once and loaded into an approximate-nearest-neighbour index (HNSW and relatives). At request time you only run the user tower — one forward pass — to get the user vector, then ask the index for its nearest item vectors. Scoring is a dot product, not a model call, so millions of items are effectively scored by geometry in a few milliseconds. It is the same retrieval machinery used in RAG systems.
What is a feedback loop (and position bias) in recommendation systems?: A feedback loop is the self-reinforcing bias that arises because the ranker is trained on logged behaviour — but users can only click items the current system chose to show them. The training data is a sample of the old policy's choices, not of the world. Two biases compound: presentation bias, where an item never shown can never be clicked so popular shown items accrue ever more evidence and the rich get richer; and position bias, where an item shown at the top gets clicked more because of its position, so the model mistakes position for quality. Left untreated the loop collapses the catalog toward a handful of hits and produces filter bubbles. The defences are to log the propensity (how likely each item was to be shown) so you can inverse-weight it, to deliberately explore uncertain items, and to log context honestly so the bias is measurable.
How do LLMs change recommendation systems?: They change what goes inside the funnel stages, not the funnel itself — nothing about a language model lets you score ten million items per request. Three shifts are real. Semantic retrieval uses LLM-quality embeddings so retrieval understands meaning rather than just co-occurrence; it slots into the existing two-tower-plus-ANN layer as one more source. LLM-as-ranker reads the user context and a candidate together like a cross-encoder and judges relevance far more sharply than a dot product, but it is far more expensive per item, so it lives at the narrow end of the funnel re-scoring the top handful, never the corpus. Generative recommendation, where a model emits an item's "semantic ID" directly the way it generates the next token, is genuinely new but still unsettled research, not the default. The hype to reject is "just ask an LLM to recommend from the catalog" — it cannot hold millions of items in context, cannot answer in the latency budget, and hallucinates items that do not exist.
What is train/serve skew in a recommendation ranker?: Train/serve skew is when the feature values a ranker was trained on differ from the values it is served in production, so a model that looked brilliant offline degrades live with nothing in the logs to point at. It happens when a feature like "user's clicks in the last hour" is computed one way by the batch job that builds the training set and a subtly different way by the service that computes it live at request time — a different time window, different rounding, or a value that had not settled yet. The model trained on one number and is served another; nothing errors, accuracy just quietly decays. The fix is to compute each feature once from a single definition and feed both training and serving from it, which is the core job of a feature store. It is one of the most common ways a recommender fails in production, which is why it is worth naming explicitly in an interview.

Design a Recommendation System (in the LLM Era)

Questions

Related explainers