---
topic: system-design
author: Crashtech Editorial
published: Nov 17, 2026 · read: 4 min
---

Inside X Recommendation Algorithm: In-Network Candidates, Out-of-Network Graphs, and Grok Ranking

How X (Twitter) ranks billions of posts in milliseconds: candidate sourcing, SimClusters, graph traversal, and heavy Grok transformer ranking models.

– –

Every time a user opens the X (formerly Twitter) mobile application and taps the “For You” tab, the system executes one of the most demanding real-time computing tasks on the internet.

From a corpus of over 500 million daily tweets, the recommendation pipeline must select, filter, score, and rank a personalized stream of roughly 150 candidate posts.

It must evaluate social connections, topical interests, media formats, and user feedback signals, and render the final feed in less than 150 milliseconds.

When X open-sourced the code for its recommendation algorithm, the global engineering community gained rare, unvarnished visibility into the internal plumbing of a hyper-scale discovery engine.

More importantly, the architecture has undergone a radical modern transformation: replacing hundreds of hand-tuned procedural ranking heuristics with a unified Grok-based Transformer model.

+-------------------------------------------------------------------------------+
|                       THE X RECOMMENDATION SYSTEM PIPELINE                    |
+-------------------------------------------------------------------------------+

[User opens X app: For You Tab Request]
       |
       v
+-------------------------------------------------------------------------------+
| STAGE 1: CANDIDATE SOURCING (Extract ~1,500 candidates from 500M pool)         |
+-------------------------------------------------------------------------------+
| In-Network (~50%): Earlybird Search over User Graph & Follows                 |
| Out-of-Network (~50%): SimClusters + RealGraph + GraphJet Community Random Walk|
+-------------------------------------------------------------------------------+
       |
       v
+-------------------------------------------------------------------------------+
| STAGE 2: HEURISTIC FILTERING & VISIBILITY FILTER (Prunes to ~800 posts)       |
+-------------------------------------------------------------------------------+
| - Blocklists, mutes, NSFW filters, duplicate deduplication                    |
| - Feedback fatigue penalties (Too many posts from single author)              |
+-------------------------------------------------------------------------------+
       |
       v
+-------------------------------------------------------------------------------+
| STAGE 3: HEAVY RANKING (Grok Transformer Neural Network)                      |
+-------------------------------------------------------------------------------+
| Evaluates P(Like), P(Retweet), P(Reply), P(Report), P(VideoView)              |
| Computes weighted total engagement score per candidate post                   |
+-------------------------------------------------------------------------------+
       |
       v
+-------------------------------------------------------------------------------+
| STAGE 4: MIXING & DIVERSITY FILTERING (Final 150 posts rendered in < 150ms)   |
+-------------------------------------------------------------------------------+
| Interleaves ads, in-network/out-of-network balance, author diversity          |

1. Stage 1: Candidate Sourcing (Finding the 1,500)

At any given second, the active tweet corpus contains billions of records. Evaluating a heavy deep learning transformer against 500 million posts for every user request is computationally impossible.

Stage 1 rapidly reduces 500 million posts down to a manageable set of ~1,500 candidate tweets using two distinct pipelines:

+-------------------------------------------------------------------------------+
|                       IN-NETWORK vs OUT-OF-NETWORK SOURCING                   |
+-------------------------------------------------------------------------------+

In-Network Candidates (~50%):
  - Fetches recent tweets from accounts the user explicitly follows.
  - Storage: In-memory inverted index (Earlybird / Lucene fork).
  - Maximizes social relevance and immediate recency.

Out-of-Network Candidates (~50%):
  - Discovers tweets from accounts the user DOES NOT follow.
  - Algorithms: SimClusters (Community Topic Graphs) & GraphJet.
  - Maximizes discovery, serendipity, and viral global content.

1. SimClusters: Community Embeddings

X partitions users into thousands of distinct communities using Matrix Factorization over the follow graph (e.g., Pop Music Fans, Rust Programmers, Venture Capitalists).

Tweets are embedded into these community spaces. If 20 people in the “Distributed Systems Engineers” community favorite a tweet, that tweet is automatically surfaced as a candidate for other members of that community, even if they do not follow the author.

2. GraphJet: Real-Time Bipartite Graph Walks

GraphJet maintains a volatile in-memory bipartite graph of users and engagements (likes, retweets) updated in real time.

By executing randomized graph traversals (“Users who liked what you liked also engaged with Post X”), GraphJet captures emerging viral trends within seconds of publication.


2. Stage 2: The Visibility Filtering Gate

Before expensive neural scoring takes place, candidate posts traverse the Visibility Filtering Gate. This deterministic layer prunes roughly 40% of candidates:

  • Safety & Blocklists: Removes posts from authors the user has blocked, muted, or reported.
  • Content Moderation: Drops illegal content, spam, and unlabelled graphic violence.
  • De-duplication: Filters out duplicate retweets and previously seen tweets stored in the user’s impression cache.
  • Author Fatigue Limiting: Caps the maximum number of consecutive candidates from a single account.

3. Stage 3: The Heavy Ranking Model (From Hand-Tuned Heuristics to Grok)

In the original open-source release, candidates were scored by a MaskNet deep neural network using hand-crafted logistic regression feature combinations.

In the modern architecture, X replaced these procedural heuristics with a Grok-based Transformer Ranking Engine.

Multi-Task Prediction

The ranking model does not predict a single “good or bad” score. It simultaneously outputs independent probability estimates for multiple discrete user interactions:

+-----------------------------------------------------------------------+
|                      ENGAGEMENT PROBABILITY PREDICTIONS               |
+-----------------------------------------------------------------------+

Target Interaction Metric               Weight in Final Utility Function
-------------------------------------------------------------------------
P(Favorite / Like)                      + 0.5
P(Retweet / Repost)                     + 1.0
P(Reply with User Engagement)           + 13.5 (High Conversational Value)
P(Click Profile / Author Follow)        + 4.0
P(Video Watch Time > 50%)               + 2.0
P(Report Tweet / "Show Less Often")     - 369.0 (Massive Negative Penalty!)

The Unified Utility Equation

The final score is a weighted linear combination of these probabilities:

FinalScore = sum(W_i * P(Action_i))

Notice the immense negative penalty assigned to negative feedback signals. A 1% probability that a user reports a post can mathematically erase hundreds of predicted likes, driving the tweet out of the feed entirely.


4. Stage 4: Mixing, Diversity, and Dynamic Insertion

Having scored the remaining candidates, the pipeline must assemble the final timeline viewable on the user’s glass:

[Ranked Candidates: Ranked 1 to 800]
       |
       v
[Timeline Mixer & Sequencer]
       |-- Enforces 50/50 balance between In-Network and Out-of-Network
       |-- Injects Sponsored Advertisements (Targeted Ad Auction Engine)
       |-- Injects "Who to Follow" Social Modules
       |-- Enforces Author Diversity (Maximum 2 tweets per author per screen)
       v
[Rendered Stream: Top 150 Posts flushed to mobile client in < 150ms]

Architectural Comparison Matrix

System ComponentLegacy Procedural AlgorithmModern Grok-Transformer Pipeline
Out-of-Network SourcingHeuristic bipartite graph random walksDense vector embeddings + SimClusters
Ranking CoreHeavy logistic regression + MaskNetUnified Grok Transformer architecture
Feature Engineering1,000+ hand-crafted scalar featuresEnd-to-end token and interaction embeddings
Negative SignalsSimple rule-based downrankingHeavy dynamic loss penalty ($W = -369.0$)
Serving SLASub-200ms P99 latencySub-150ms P99 latency via GPU batching

Summary and Key Takeaways

  1. Deconstruct hyper-scale ranking into stages: Reduce 500 million items to 1,500 candidates via lightweight graph retrieval before running heavy neural ranking models.
  2. Balance familiarity with discovery: Blend in-network social graphs with out-of-network community clusters to maintain connection while fueling discovery.
  3. Multi-task prediction is mandatory: Model distinct engagement probabilities (likes, retweets, replies, video views) rather than a single monolithic click-through rate.
  4. Penalize negative feedback aggressively: A small probability of negative user action (reports, mutes) must carry an order-of-magnitude higher weight than positive engagements.
  5. Transformers replace hand-crafted heuristics: Modern discovery engines favor unified transformer architectures capable of learning cross-modal relationships over procedural rule engines.
Advertisement

/* Comments */