---
topic: system-design
author: Crashtech Editorial
published: Aug 10, 2026 · read: 4 min
---

Fixing Clickbait at Retrieval: Why Meta, LinkedIn, and YouTube Chose Meaning Over Engagement

Why engagement-based candidate retrieval fuels clickbait, and how two-tower embeddings and semantic IDs fix feeds at the source.

– –

Every major content platform has fought the same war against clickbait. A creator posts an article with an emotionally charged, sensationalized headline: “The One Habit That Destroys Your Brain After Age 30.”

Users impulsively click the link. Within ten seconds, they realize the article contains zero scientific substance, feel deceived, and immediately hit the back button.

For over a decade, social platforms treated this as a moderation and post-ranking problem:

  • Train specialized classification models to detect sensationalist words in titles.
  • Penalize posts whose bounce rates exceed 80%.
  • Deploy human review teams to manually flag abusive publishers.

None of these downstream fixes solved the problem. The reason is structural: Clickbait is not a moderation failure; it is a candidate retrieval failure.

When a recommendation engine relies on raw engagement metrics during the initial retrieval stage, clickbait items dominate the candidate pool before deeper ranking models even get to look at them. Fixing clickbait required redesigning the feed from engagement-based heuristic matching to semantic representation.

+--------------------------------------------------------------------------+
|                  THE TWO-STAGE FEED RECOMMENDATION FUNNEL               |
|                                                                          |
|  [Global Corpus: 500,000,000 Posts]                                      |
|                 |                                                        |
|                 v                                                        |
|  STAGE 1: Candidate Retrieval (Latency: <15ms)                           |
|  Filters corpus down to ~1,000 candidate items.                          |
|  *IF BASED ON CLICKS: Clickbait floods the candidates. Quality is lost.*|
|                 |                                                        |
|                 v                                                        |
|  STAGE 2: Heavy Multi-Task Ranking (Latency: ~50ms)                      |
|  Deep cross-attention transformer scores top 1,000 -> Emits Top 20.      |
+--------------------------------------------------------------------------+

Why Heuristic Retrieval Fuels Sensationalism

In a production feed (LinkedIn, Meta, YouTube, TikTok), the engine cannot evaluate deep neural ranking networks across hundreds of millions of public posts in real time.

The system is split into two stages:

  1. Candidate Retrieval (Candidate Generation): Must evaluate the entire catalog in under 15 milliseconds. Historically, this relied on inverted indexes, collaborative filtering (Matrix Factorization), and popularity heuristics: What did the user’s friends click? What is trending in their geographic region?
  2. Heavy Ranking: Evaluates complex multi-task neural networks across the top 1,000 candidates to select the 20 posts displayed on screen.

Because Stage 1 had to be extremely computationally cheap, it used raw click-through rate (CTR) as its primary proxy for relevance. Clickbait exploits this proxy perfectly: it triggers high early clicks, flooding the candidate pool. Once the top 1,000 items are saturated with clickbait, the Stage 2 ranker is forced to choose between twenty slightly different flavors of low-quality bait.

The Semantic Shift: Three Industry Architectures

To eliminate clickbait at the source, engineering teams at LinkedIn, Meta, and YouTube replaced engagement-heuristic retrieval with Semantic Retrieval, forcing the candidate generation stage to evaluate semantic meaning and user intent rather than raw clicks.

[LinkedIn]: Consolidates 5 retrieval stacks into 1 LLM Dual-Encoder (Qwen Backbone)
[Meta]    : Funnel of specialized models feeding a Multi-Objective Value Model (MOVM)
[YouTube] : Generative Retrieval emitting content Semantic IDs without an index

1. LinkedIn: Consolidated Two-Tower Semantic Encoder

Historically, LinkedIn maintained five independent retrieval engines: one for connections, one for hashtags, one for companies, one for search keywords, and one for viral posts.

LinkedIn replaced this fragmented architecture with a single unified Two-Tower LLM Dual Encoder:

  • Query Tower (User Context): Embeds the member’s profile, industry, recent dwell history, and professional skills into a dense 256-dimensional vector.
  • Item Tower (Post Content): Pre-computes embeddings for posts using a fine-tuned language model that ingests the full body text, author reputation, and article citations (not just the headline).
  • Because vectors represent semantic alignment between the user’s professional domain and the substantive concepts inside the post, sensationalist posts lacking depth fail vector similarity lookup in the HNSW index, never reaching the candidate pool.

2. Meta: The Multi-Objective Value Model (MOVM)

Meta’s feed architecture operates at an incomprehensible scale: billions of active users generating tens of billions of feed impressions daily.

Meta tackled clickbait by replacing single-metric CTR optimization with Multi-Objective Value Models (MOVM):

Value = w1 * P(Click) + w2 * P(Long Dwell > 60s) + w3 * P(Share) - w4 * P(Immediate Bounce) - w5 * P(Hide)
  • A click followed by an immediate exit within 10 seconds is assigned a massive negative value weight ($w_4$).
  • Posts that accumulate high initial clicks but heavy negative weights are dynamically excised from edge candidate retrieval caches within seconds of publication.

3. YouTube: Generative Retrieval via Semantic IDs

YouTube pioneered the most radical architectural departure: Generative Retrieval.

Traditional retrieval stores embeddings in an Approximate Nearest Neighbor (ANN) vector database (like Faiss or HNSW) and performs cosine similarity queries.

YouTube experimented with eliminating the vector index entirely:

  • Content is tokenized into hierarchical Semantic IDs using Residual Vector Quantization (RVQ), e.g., [Cat: Tech, Sub: Distributed, Specific: Consensus, ID: 8829].
  • A sequence-to-sequence transformer model ingests the user’s viewing history and directly predicts the Semantic ID tokens of the next video they will find satisfying.
  • Because the model generates discrete semantic tokens rather than matching popularity scores, it predicts substantive topic transitions rather than sensationalist thumbnail traps.

Tradeoffs: Semantic Cold-Starts vs. Stereotyping

Shifting retrieval to semantic representations eliminates clickbait, but introduces new engineering challenges:

DimensionEngagement-Based RetrievalSemantic Vector Retrieval
Clickbait VulnerabilityExtremely high (Exploits click proxy)Very low (Ranks on content meaning)
New Content Cold-StartPoor (Requires initial impressions)Excellent (Embeds instantly on write)
The Stereotype TrapDiverse (Follows viral network trends)Over-specialized (Pigeonholes users into narrow topic bubbles)
Serving CostLow (Inverted index integer lookup)High (Vector dot products / GPU caches)

Semantic retrieval does not magically eradicate sensationalism from the internet, but it strips clickbait of its structural superpower: the ability to hijack candidate generation. When your retrieval layer understands content meaning, sensationalism loses its shortcut to the top of the feed.

Advertisement

/* Comments */