---
topic: hardware
author: Crashtech Editorial
published: Aug 15, 2026 · read: 4 min
---

TPU 8t and TPU 8i: How Google Custom AI Chips Split Training and Inference Hardware

Comparing Google's eighth-generation custom ASICs: throughput-oriented TPU 8t training pods versus low-latency TPU 8i serving.

– –

When modern software engineers consider artificial intelligence accelerators, NVIDIA GPUs dominate the conversation. But inside Google’s global data centers, an alternate silicon reality has quietly powered production workloads for over a decade: the Tensor Processing Unit (TPU).

From Google Search rankBrain and YouTube recommendations to the training and serving of Gemini, Google relies on custom Application-Specific Integrated Circuits (ASICs).

Graphics Processing Units (GPUs) were originally engineered for rendering 3D polygons, rasterizing triangles, and executing general-purpose SIMD instructions. TPUs, by contrast, were designed from first principles for a single mathematical operation: dense matrix multiplication ($C = A \times B + C$).

With the rollout of its eighth-generation silicon, Google made a decisive architectural split: bifurcating its fleet into two specialized silicon variants: TPU 8t (Training) and TPU 8i (Inference).

+--------------------------------------------------------------------------+
|                  THE DUAL SILICON SPECIALIZATION                         |
|                                                                          |
|  [TPU 8t: Training Specialized]        [TPU 8i: Inference Specialized]   |
|  - Maximize Bfloat16 FLOPs/Watt        - Minimize Time-to-First-Token    |
|  - Optical Circuit Switching (OCS)     - High Batch-1 Memory Bandwidth   |
|  - Liquid-cooled 10,000-chip pods      - Cost-effective rack density     |
+--------------------------------------------------------------------------+

The Mathematical Foundation: Systolic Arrays

To understand why a dedicated TPU outperforms a general-purpose processor, examine how computers multiply matrices.

In a traditional CPU or GPU, every arithmetic operation requires:

  1. Fetching instruction from cache.
  2. Loading Operand A from register memory.
  3. Loading Operand B from register memory.
  4. Passing operands through the Arithmetic Logic Unit (ALU).
  5. Writing the intermediate result back to a register.

In deep neural networks with billions of weights, memory access consumes 100 times more energy and latency than the arithmetic computation itself.

TPUs bypass this bottleneck using a Systolic Array:

        Weights pre-loaded into stationary grid cells:
        [ W_11 ] ---> [ W_12 ] ---> [ W_13 ]
           |             |             |
           v             v             v
Inputs flow from left:
Input X1 ---> [*] ----------> [*] ----------> [*] ---> Accumulated Output Y1
Input X2 ---> [*] ----------> [*] ----------> [*] ---> Accumulated Output Y2

In a systolic array:

  • Matrix weights are pre-loaded into stationary physical multipliers across a two-dimensional grid (typically $128 \times 128$ or $256 \times 256$ cells).
  • Input activations stream continuously from the left.
  • Accumulated values stream downward like blood pumping through a heart (hence “systolic”).
  • Intermediate products pass directly from one cell to its neighbor without touching registers or high-bandwidth memory.

This eliminates register file read-write energy entirely, allowing a single chip to achieve tens of teraflops per watt.

Training vs. Inference: Why Google Split the Silicon

Historically, earlier TPU generations (v2, v3, v4) attempted to balance both training and inference on the same chip. But as model sizes scaled from 10 billion to over 1 trillion parameters, the physical constraints of training and inference diverged completely:

DimensionTraining Workload (TPU 8t)Inference Workload (TPU 8i)
Primary MetricSustained TeraFLOPS throughputLatency per token (TTFT and inter-token)
Batch SizeMassive (Thousands of sequences)Small (Batch size 1 to 32)
Interconnect NeedMassive chip-to-chip bandwidth (Torus)Low inter-chip overhead (Host-to-chip)
Memory FootprintHolds weights, gradients, optimizer statesHolds model weights + dynamic KV cache
PrecisionMixed FP32/BF16 with dynamic scalingAggressive FP8 / INT4 quantization

TPU 8t: The Scaled Training Engine

Training frontier foundation models requires distributing billions of parameters across thousands of interconnected chips using tensor and pipeline parallelism:

  • Optical Circuit Switching (OCS): TPU 8t pods connect up to 9,216 chips in a dynamic 3D Torus network using reconfigurable fiber-optic switches. If a node fails, optical switches dynamically re-route around the damaged link in microseconds without dropping the global training job.
  • Extreme Liquid Cooling: To sustain peak Bfloat16 matrix throughput without thermal throttling, TPU 8t blades are direct-to-chip liquid cooled.

TPU 8i: The Low-Latency Inference Engine

Once a model is trained, the financial cost shifts to serving millions of live user queries:

  • Optimized for Small Batch Sizes: In real-time chat, users wait for each token. TPU 8i optimizes its matrix units and SRAM caches to achieve maximum throughput at batch sizes of 1 to 4.
  • Axion ARM Host Integration: TPU 8i servers are paired natively with Google’s custom Axion ARM CPUs, minimizing host-to-device PCIe transfer overhead.
  • Air-Cooled Rack Density: TPU 8i chips operate at lower thermal design power (TDP), allowing them to fit into standard air-cooled data center racks worldwide, democratizing inference scale.

The Software Unified Moat: XLA and JAX

Building custom silicon is useless without an optimizing compiler. Google’s enduring advantage is not merely the physical ASICs; it is XLA (Accelerated Linear Algebra) and the JAX ecosystem:

  • Python code written in JAX or PyTorch compiles into a unified intermediate representation (HLO).
  • The XLA compiler analyzes the execution graph and performs automated kernel fusion: combining matrix multiplication, bias addition, and activation functions into a single compiled binary block.
  • Developers write code once: the same program runs seamlessly across TPU 8t training pods and scales down to TPU 8i inference instances with zero manual CUDA optimization.

Custom silicon specialization is the endgame of AI infrastructure: hardware specifically contoured to the mathematical shape of the workload.

Advertisement

/* Comments */