---
topic: ai-engineering
author: Crashtech Editorial
published: Aug 4, 2026 · read: 4 min
---

The KV Cache Is the Bill: Why Long-Context Inference Gets Expensive and Which Levers Actually Cut It

Why attention scales quadratically in memory, how the KV cache dominates inference cost, and how PagedAttention, GQA, and MLA cut it.

– –

When engineers calculate the hardware needed to serve an LLM, their first instinct is to count parameters. A 70-billion-parameter model loaded in 16-bit precision requires roughly 140 gigabytes of VRAM. An NVIDIA H100 GPU holds 80 gigabytes, so two H100s should comfortably run the model.

In production, that cluster crashes with an Out of Memory (OOM) error on its fifth concurrent request.

The reason is the Key-Value (KV) Cache. During autoregressive generation, calculating attention requires multiplying the query vector of the new token against the key and value vectors of every preceding token in the sequence. To avoid recomputing these vectors on every token generation step, the inference engine caches them in GPU memory.

As context windows expand from 4,000 tokens to 128,000 and 1,000,000 tokens, the KV cache rapidly outgrows the model weights themselves. The KV cache is not an implementation detail; it is the fundamental economic bottleneck of long-context AI.

+-------------------------------------------------------------+
|             THE MEMORY FOOTPRINT REVERSAL                   |
|                                                             |
| Short Context (2K tokens):                                  |
| [ Model Weights: 140 GB ] >>> [ KV Cache: 2.5 GB ]          |
|                                                             |
| Long Context (128K tokens, batch size 8):                   |
| [ Model Weights: 140 GB ] <<< [ KV Cache: 320 GB! ]         |
+-------------------------------------------------------------+

The Mathematical Growth of the KV Cache

To understand why memory escalates so aggressively, examine the standard KV cache memory equation per active sequence:

Memory (bytes) = 2 * 2 * n_layers * n_heads * d_head * L * b

Where:

  • The first $2$ accounts for two tensors: Keys and Values.
  • The second $2$ represents 16-bit precision (2 bytes per FP16 element).
  • n_layers is the number of transformer layers.
  • n_heads is the number of attention heads.
  • d_head is the head dimension.
  • $L$ is sequence length (context window).
  • $b$ is the concurrent batch size.

For a standard 70B model (n_layers = 80, n_heads = 64, d_head = 128):

  • A single sequence at 4,000 tokens consumes ~650 MB of VRAM.
  • A single sequence at 128,000 tokens consumes ~20.8 GB of VRAM.
  • A batch of just 8 concurrent users running 128K context demands 166 GB of VRAM solely for the KV cache, exceeding two H100 GPUs before counting model weights!

The Fragmentation Disaster: How vLLM and PagedAttention Fixed It

Prior to 2023, inference engines allocated KV cache memory using contiguous memory buffers. Because an engine could not predict whether a user prompt would generate 10 tokens or 2,048 tokens, it had to pre-allocate a contiguous memory block for the maximum possible sequence length.

This created massive internal and external memory fragmentation:

  • Up to 60% to 80% of GPU memory sat completely empty, reserved for tokens that were never generated.
  • Other requests were queued or rejected due to artificial memory exhaustion.

In 2023, researchers at UC Berkeley introduced PagedAttention (the core engine of vLLM), drawing inspiration from virtual memory and paging in classic operating systems:

Physical GPU VRAM:
[Block 0: Tokens 0-15] -> [Block 1: Tokens 16-31] -> [Block 2: Tokens 32-47]
       ^                        ^                         ^
       |                        |                         |
Page Table (Req 1):       Page Table (Req 2):       Page Table (Shared System Prompt):
[0] -> Block 0            [0] -> Block 4            [0] -> Block 0 (Zero-copy copy-on-write!)
[1] -> Block 1            [1] -> Block 5
  • Instead of contiguous arrays, the KV cache is chopped into fixed-size blocks (typically 16 or 32 tokens).
  • A virtual page table maps logical sequence tokens to non-contiguous physical GPU memory blocks.
  • Blocks are allocated dynamically on-demand as tokens are generated.
  • Memory waste dropped from 70% to under 4%, instantly tripling serving throughput on existing GPU hardware.

The Architectural Levers: Shrinking the Cache Size

PagedAttention eliminates wasted space, but it does not change the mathematical volume of data generated. To push context windows further, researchers redesigned the underlying transformer attention mechanism.

       MHA (Multi-Head Attention)        GQA (Grouped-Query Attention)         MLA (Multi-Head Latent)
         Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8             Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8            Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8
          |  |  |  |  |  |  |  |              \  /   \  /   \  /   \  /                  \   |   /
         K1 K2 K3 K4 K5 K6 K7 K8               K1      K2      K3      K4               [ Compressed Latent ]
         V1 V2 V3 V4 V5 V6 V7 V8               V1      V2      V3      V4               [ Vector c_KV (512d)]
         (1 KV Head per Q Head)                (1 KV Head per 4 Q Heads)               (93% KV Cache Reduction)

1. Grouped-Query Attention (GQA)

In standard Multi-Head Attention (MHA), every query head has its own dedicated key and value head.

  • Grouped-Query Attention (used in Llama 3 and Mistral) forces multiple query heads (e.g., 8 heads) to share a single key-value head group.
  • If 8 query heads share 1 KV head, the KV cache memory footprint shrinks by an immediate 8x (87.5% reduction) with virtually zero loss in benchmark reasoning performance.

2. Multi-Head Latent Attention (DeepSeek MLA)

DeepSeek-V2 and V3 introduced a revolutionary compression architecture: Multi-Head Latent Attention (MLA).

  • Instead of caching full key and value matrices for every head, MLA projects the keys and values into a low-dimensional compressed latent space (c_KV) of only 512 dimensions.
  • The inference engine stores only this compact latent vector in the KV cache.
  • During attention calculation, the latent vector is decompressed on the fly using up-projection matrix weights stored in fast GPU SRAM.
  • Result: A massive 93% reduction in KV cache memory compared to standard MHA. This architectural breakthrough is the primary reason DeepSeek can serve 128K context models at a fraction of Western hyperscaler pricing.

3. KV Cache Quantization (FP8 and INT4)

By default, KV caches are stored in 16-bit floating point (FP16/BF16).

  • Quantizing cached KV tensors to FP8 halves memory consumption and doubles memory bandwidth speed with negligible perplexity degradation.
  • Aggressive sub-4-bit quantization (KIVI, INT4) offers 4x compression, though it requires outlier-aware calibration to avoid degrading long-range needle retrieval.

The Serving Hierarchy

TechniqueWhere AppliedMemory ReductionLatency ImpactTradeoff
PagedAttentionSoftware RuntimeEliminates ~70% wasteFaster (Zero waste)Requires paging table runtime
GQAModel Architecture8x reductionFaster decodeMust be trained into model
DeepSeek MLAModel Architecture14x reductionDrastically fasterRequires custom CUDA kernels
FP8 QuantizationRuntime / Weights2x reduction2x memory throughputMinor precision noise

When architecting long-context AI systems, remember: model weights set the floor, but the KV cache sets the ceiling. Managing your KV memory footprint is the difference between an application that scales profitably and one that collapses under its own token bill.

Advertisement

/* Comments */