---
topic: engineering
author: Crashtech Editorial
published: Nov 17, 2026 · read: 3 min
---

How Snapchat Serves a Billion Predictions Per Second

Inside Bento, Snapchat's ML ranking engine that fans out candidate scoring to thousands of model calls in 100 milliseconds across 474M active users.

– –

When 474 million daily active users open Snapchat, the platform has roughly 100 milliseconds to choose what content to show. In that brief window across Spotlight, Discover, Stories, and sponsored ads, Snapchat’s recommendation infrastructure executes a massive fan-out: retrieving hundreds of candidate items from a pool of millions, pulling real-time contextual features, running deep learning ranking models, and returning an ordered feed.

At peak traffic, this engine evaluates more than a billion candidate predictions every single second. Snap engineered a specialized platform named Bento to handle this asymmetric workload without blowing past latency deadlines or cloud budgets.

The Asymmetry of Ranking Workloads

Standard microservices follow symmetric request patterns: an incoming RPC requires fetching a user profile, checking a database, and returning a response payload of comparable size. Ranking services, by contrast, are fundamentally asymmetric:

1 User Request -> 1,000 Candidates -> 1,000 (User, Item) Scoring Pairs -> Top 20 Ranked Items

Each scoring pair combines static item metadata (video duration, creator ID, category embeddings) with dynamic user signals (recent watch time, category affinities, skips, geolocation, device type). Scoring 1,000 candidates with a multi-layer neural network requires billions of floating-point operations per user request.

If candidate scoring is routed over standard internal gRPC networks with naively serialized payloads, network interfaces saturate almost instantly. At Snapchat’s scale, serializing and deserializing feature dictionaries across network boundaries easily consumes more CPU cycles than running the actual matrix multiplications.

Architecture of the Bento Platform

Snapchat separated its ranking stack into two distinct pipelines: offline training and online low-latency inference.

+-----------------------------------------------------------------------+
|                       Snapchat Bento Architecture                     |
+-----------------------------------------------------------------------+
|                                                                       |
|  [ Client App ]                                                       |
|        | (gRPC, < 100ms budget)                                       |
|        v                                                              |
|  [ API Gateway / Orchestrator ]                                       |
|        |                                                              |
|        +---> [ Candidate Retrieval: ANN Vector Search ] (~1,000 items)|
|        |                                                              |
|        +---> [ Low-Latency Feature Store: Redis / RocksDB ]           |
|        |          (User State, Interaction History, Graph Embeddings) |
|        v                                                              |
|  [ Bento Prediction Service ]                                         |
|        |                                                              |
|        +---> Zero-Copy Shared Memory Buffer                           |
|        +---> CPU Feature Transformation & Embedding Lookups           |
|        +---> GPU Tensor Cores: Fused Kernel Candidate Scoring         |
|        v                                                              |
|  [ Calibrated Scores & Diversification ]                              |
|        |                                                              |
|        v                                                              |
|  [ Top 20 Ranked Feed -> Return to User ]                             |
+-----------------------------------------------------------------------+

1. Training and Graph Partitioning

Offline training runs on Kubeflow clusters executing automated training pipelines. Snap decoupled ranking models into three clean tiers:

  • Core Framework: Standard shared training loops, evaluation harnesses, and distributed tensor primitives.
  • Team Code: Custom neural network architectures and loss functions authored by individual product teams (Spotlight vs. Ads).
  • Configuration (YAML): Feature definitions, learning rates, embedding table dimensions, and export targets.

At export time, the compiler performs a critical graph split: sparse embedding tables and categorical string transformations remain pinned to high-memory host CPUs, while dense deep neural layers are compiled into optimized TensorRT engines executing on GPU clusters.

2. Feature Storage and Zero-Copy Serialization

Fetching 50 features for 1,000 candidate items produces 50,000 feature lookups per query. To survive this load, Bento structures features into dense flat arrays rather than nested JSON or Protobuf maps.

Features are divided into three tiers:

  1. Static Item Features: Pre-computed embeddings and creator metadata cached directly in local memory on inference host instances using shared memory mappings.
  2. Real-time User Features: Rolling counters (clicks in last 5 minutes, skips in last 60 seconds) served from distributed in-memory key-value stores with sub-millisecond p99 latencies.
  3. Contextual Features: Network connection type, local time, battery status, and app version extracted directly from the request header.

By using flat memory buffers and zero-copy deserialization, Bento bypasses object allocation overhead in the hot serving path.

Mitigating the 100ms Latency Budget

To maintain a strict 100ms deadline, Bento applies hard architectural deadlines and early-exit heuristics:

Stage Budget Breakdown:
- Network ingress and parsing: 10ms
- Candidate retrieval (ANN): 20ms
- Feature store lookups: 25ms
- Model scoring (Bento GPU): 30ms
- Diversification and reranking: 10ms
- Network egress: 5ms
Total: 100ms

When candidate retrieval or feature lookups threaten the deadline, Bento triggers adaptive degradation: reducing the candidate pool from 1,000 to 400 items, or falling back to a lightweight linear model if GPU queue depth crosses safe thresholds.

Key Engineering Takeaways

  • Feature serialization is the silent killer: In high-concurrency ranking systems, Protobuf or JSON decoding overhead often dwarfs model execution time. Use flat, zero-copy buffers.
  • Split sparse and dense computation: Dense matrix multiplications belong on GPU tensor cores; sparse embedding table lookups belong in high-bandwidth host memory.
  • Enforce strict time budgeting: Every stage in the fan-out must have an explicit timeout, accompanied by graceful fallback paths to lightweight models.
Advertisement

/* Comments */