---
topic: engineering
author: Crashtech Editorial
published: Nov 17, 2026 · read: 3 min
---

A Complete Guide to AI Inference Engineering: Prefill vs Decode

Why LLM inference splits into compute-bound prefill and memory-bound decode. Master PagedAttention, KV-caching, speculative decoding, and vLLM.

– –

When software engineers first deploy Large Language Models on cloud GPUs, they are often stunned by performance anomalies: A high-end GPU capable of hundreds of teraflops achieves barely 30 tokens per second for a single user, with GPU compute utilization hovering under 15%.

The reason lies in a fundamental duality of autoregressive transformer models: Inference is not one single workload; it is two completely opposite computational phases executing back-to-back on the same hardware.

Mastering inference engineering requires understanding the Prefill-Decode Split and deploying techniques like PagedAttention, continuous batching, and speculative decoding to saturate hardware memory bandwidth.

The Core Duality: Prefill vs Decode

+-----------------------------------------------------------------------+
|                       The Prefill-Decode Split                        |
+---------------------------+-------------------------------------------+
| Dimension                 | Prefill Phase (Prompt Processing)         |
+---------------------------+-------------------------------------------+
| Primary Bottleneck        | Compute-Bound (Tensor Cores / FLOPs)      |
| Input -> Output           | N Prompt Tokens In -> Key-Value Cache Out |
| Parallelism               | Highly Parallel (Matrix x Matrix)         |
| Primary Metric            | Time to First Token (TTFT)                |
+---------------------------+-------------------------------------------+
| Dimension                 | Decode Phase (Token Generation)           |
+---------------------------+-------------------------------------------+
| Primary Bottleneck        | Memory-Bandwidth Bound (HBM Bus Transfer) |
| Input -> Output           | 1 Token In -> 1 Token Out                 |
| Parallelism               | Sequential Autoregressive (Matrix x Vector)|
| Primary Metric            | Inter-Token Latency (Tokens per Second)   |
+---------------------------+-------------------------------------------+

Phase 1: Prefill (Compute-Bound)

When a user submits a 2,000-token prompt, the GPU processes all 2,000 tokens concurrently. The model multiplies the input embeddings across transformer weights in large, dense matrix-matrix multiplications (GEMM).

  • Hardware behavior: Tensor cores are fully saturated. Computation is compute-bound.
  • Output: The model computes and stores the Key and Value vectors for all 2,000 tokens into GPU High-Bandwidth Memory (HBM). This is the KV Cache.

Phase 2: Decode (Memory-Bandwidth Bound)

To generate the first output token, the model needs only the newest token and the existing KV cache. To generate token 2,001, the entire 70-billion-parameter model weight matrix must be fetched from HBM into the compute cores:

Memory Fetched per Single Token: 70 Billion Parameters × 2 Bytes (FP16) ≈ 140 Gigabytes!

The GPU spends 98% of its time waiting for memory buses to stream 140 GB of model weights across High-Bandwidth Memory just to perform a few matrix-vector multiplications for one single token. The arithmetic intensity collapses.

Production Techniques to Saturate GPUs

Because single requests leave GPUs under-utilized, inference engineering focuses on six key architectural solutions:

+-----------------------------------------------------------------------+
|                 Modern Inference Optimization Stack                   |
+-----------------------------------------------------------------------+
|                                                                       |
|  [ 1. Continuous Batching (vLLM / TensorRT-LLM) ]                     |
|  - Eliminates static batch waste; injects new prompts on each token   |
|        |                                                              |
|        v                                                              |
|  [ 2. PagedAttention (Virtual Memory for KV Cache) ]                  |
|  - Eliminates 80% KV memory fragmentation; stores KV blocks in pages  |
|        |                                                              |
|        v                                                              |
|  [ 3. Prefix Caching (Radix Tree State Reuse) ]                       |
|  - Reuses pre-computed KV cache for common system prompts and schemas|
|        |                                                              |
|        v                                                              |
|  [ 4. Speculative Decoding (Draft Model Acceleration) ]               |
|  - Small model proposes 5 tokens -> Big model verifies in 1 step      |
|        |                                                              |
|        v                                                              |
|  [ 5. Disaggregated Serving (Split Prefill and Decode Clusters) ]     |
|  - Routes prefill to compute GPUs; routes decode to bandwidth GPUs    |
+-----------------------------------------------------------------------+

1. PagedAttention and Dynamic Memory Management

In naive serving frameworks, memory for the KV cache must be pre-allocated contiguously based on the maximum possible sequence length (e.g., 4,096 tokens). Because most requests generate fewer than 500 tokens, 60% to 80% of valuable GPU VRAM sits completely wasted due to internal and external fragmentation.

Inspired by virtual memory paging in operating systems, PagedAttention (pioneered by vLLM) partitions the KV cache into fixed-size physical blocks (e.g., 16 tokens per block):

  • Physical blocks do not need to be contiguous in memory.
  • Blocks are allocated on demand as new tokens generate.
  • Different requests sharing the same system prompt can share physical memory pages using copy-on-write pointers.

PagedAttention recovers virtually all wasted memory, allowing servers to increase batch sizes by 2x to 4x, directly translating to higher serving throughput.

2. Speculative Decoding

Because decoding is memory-bandwidth bound, reading model weights once to verify multiple tokens is vastly faster than reading weights five times for five individual tokens.

Speculative Decoding pairs a massive target model (e.g., 70B) with a tiny draft model (e.g., 1B):

  1. The 1B draft model quickly generates 5 candidate tokens in rapid autoregressive steps.
  2. The 70B target model evaluates all 5 candidate tokens simultaneously in a single forward pass (matrix-matrix compute mode, saturating tensor cores).
  3. Any correct draft tokens are accepted; if token 4 diverges, generation restarts from token 4.

Because draft tokens are frequently accepted, speculative decoding accelerates token generation speed by 2x to 2.5x with zero loss in mathematical precision.

3. Disaggregated Serving (Prefill-Decode Disaggregation)

In standard setups, a GPU alternates between prefill (bursts of heavy compute) and decode (trickles of memory reads). This interference creates severe latency jitter: an incoming large prompt stalls ongoing token generation for active users.

Modern hyperscale inference clusters physically separate hardware:

  • Prefill Nodes: High-compute instances (such as NVIDIA H100 with massive FP8 tensor throughput) dedicated exclusively to processing incoming prompts.
  • Decode Nodes: High-bandwidth memory instances optimized for fast token streaming.
  • The pre-computed KV cache is transferred over high-speed InfiniBand / NVLink interconnects from prefill nodes to decode nodes in milliseconds.

Key Engineering Takeaways

  • Prefill is compute-bound; Decode is memory-bound: Never evaluate LLM performance as a single number. Measure Time to First Token (TTFT) and Inter-Token Latency (ITL) separately.
  • Always use PagedAttention: Unmanaged KV caches waste over 60% of GPU memory. Frameworks like vLLM multiply effective serving capacity.
  • Speculative decoding delivers free speedups: Pairing large models with small draft models breaks the memory-bandwidth bottleneck, doubling generation speed without modifying model weights.
Advertisement

/* Comments */