CPU vs GPU vs TPU Architecture: Von Neumann Latency, SIMT Cores, and Systolic Arrays
Why hardware architecture dictates software speed: comparing CPUs, GPUs, and TPUs across control flow, SIMT parallelism, and systolic matrix multiplication.
On this page
In software engineering, developers are accustomed to treating hardware as a fungible abstraction. A compiler converts source code into machine instructions, and the operating system schedules those instructions onto silicon.
In the era of modern artificial intelligence and hyper-scale data processing, that abstraction breaks down completely.
Run a complex deep neural network on a high-end CPU, and it crawls at 2 tokens per second. Port that exact same mathematical workload to a GPU, and execution accelerates by $50 imes$. Move it onto a Google TPU, and throughput jumps another order of magnitude while power consumption drops.
Why does the exact same algorithm perform so radically differently across chips?
The answer lives in semiconductor architecture. CPUs, GPUs, and TPUs make fundamentally different physical trade-offs regarding silicon area allocation, memory access hierarchies, and instruction execution pipelines.
+-------------------------------------------------------------------------------+
| SILICON DIE AREA ALLOCATION COMPARISON |
+-------------------------------------------------------------------------------+
CPU (General Purpose, Low Latency):
+-------------------------------------------------------------------------------+
| [Complex Branch Predictor] | [Out-of-Order Control Logic] | [Giant L1/L2 Cache]|
+-------------------------------------------------------------------------------+
| [ALU Core 1] | [ALU Core 2] | [ALU Core 3] | [ALU Core 4] |
+-------------------------------------------------------------------------------+
* Philosophy: Minimize latency for a single sequential thread.
GPU (Massive Parallelism, High Throughput - SIMT):
+-------------------------------------------------------------------------------+
| [Control] | [L2 Cache] |
+-------------------------------------------------------------------------------+
| [ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU][ALU]... |
| (Thousands of tiny, simple arithmetic cores executing in lockstep warps) |
+-------------------------------------------------------------------------------+
* Philosophy: Maximize parallel floating-point throughput over thousands of threads.
TPU (Domain-Specific, Matrix Multiplication - Systolic Array):
+-------------------------------------------------------------------------------+
| [Unified Buffer] | [Matrix Multiply Unit: 2D Grid of 65,536 Multiply-Add Cells]|
+-------------------------------------------------------------------------------+
| (Data streams through neighbor cells without register-file read/write stalls!) |
+-------------------------------------------------------------------------------+
* Philosophy: Eliminate the Von Neumann memory bus bottleneck for matrix math.
1. The Central Processing Unit (CPU): The Master of Low-Latency Control Flow
The modern CPU is an engineering marvel designed around a single mandate: execute a single sequential stream of diverse, unpredictable instructions as fast as humanly possible.
Because general-purpose software (operating systems, web browsers, compilers, relational databases) is filled with complex if-else branching and pointer dereferences, CPUs dedicate over 70% of their physical silicon die area to control logic and caching rather than raw math:
- Aggressive Out-of-Order Execution Engines: Reorders instructions dynamically to keep execution pipelines saturated.
- Speculative Execution & Branch Prediction: Modern branch predictors achieve 99% accuracy guessing which path an
ifcondition will take before it evaluates. - Multi-Level Cache Hierarchies (L1, L2, L3): Massive on-die SRAM caches to insulate CPU registers from slow main RAM latency.
The CPU Bottleneck for AI
While a modern server CPU has 64 to 128 powerful cores, deep learning does not require complex branch prediction or out-of-order execution. Deep learning is billions of uniform, predictable matrix additions and multiplications.
Dedicating silicon area to giant branch predictors when you simply need to multiply two $4096 imes 4096$ matrices is an enormous waste of power and transistors.
2. The Graphics Processing Unit (GPU): Single Instruction, Multiple Threads (SIMT)
Originally engineered to render 3D polygon graphics in video games, the GPU replaced complex control flow logic with raw arithmetic density.
Instead of a few heavy cores, a modern GPU (such as the NVIDIA H100 or B200) contains thousands of lightweight arithmetic logic units (ALUs) organized into Streaming Multiprocessors (SMs).
+-------------------------------------------------------------------------------+
| NVIDIA SIMT EXECUTION (WARP LOCKSTEP) |
+-------------------------------------------------------------------------------+
[Single Instruction Issued: "MULTIPLY Matrix A by Vector B"]
|
v
+---------------------------------------------------------------+
| WARP (32 Threads executing simultaneously in lockstep) |
| Core 01: A[0] * B[0] Core 02: A[1] * B[1] |
| Core 03: A[2] * B[2] Core 04: A[3] * B[3] |
| ... Core 32: A[31] * B[31] |
+---------------------------------------------------------------+
The SIMT Execution Model
GPUs execute via Single Instruction, Multiple Threads (SIMT):
- A single instruction decoder commands a group of 32 parallel threads (a Warp).
- Every core in the warp executes the exact same mathematical instruction simultaneously on different data points.
- The Warp Divergence Trap: If code inside a GPU kernel executes an
if-elsebranch where half the threads take path A and half take path B, the GPU serializes both branches, running path A while disabling half the cores, then running path B. GPUs require uniform, branchless matrix math to achieve peak performance.
Tensor Cores
Modern GPUs incorporate dedicated Tensor Cores: specialized hardwired circuits that execute an entire $4 imes 4$ or $16 imes 16$ mixed-precision matrix multiply-and-accumulate operation ($D = A imes B + C$) in a single clock cycle.
3. The Tensor Processing Unit (TPU): The Systolic Array Revolution
While GPUs are significantly more parallel than CPUs, they still suffer from the fundamental Von Neumann memory bottleneck: for every arithmetic operation, values must be read from register files or SRAM caches and written back out.
For a matrix multiplication containing trillions of operations, shuffling intermediate activations back and forth across register buses generates extreme heat and memory bandwidth throttling.
Google solved this with the Tensor Processing Unit (TPU) by pioneering the 2D Systolic Array.
+-------------------------------------------------------------------------------+
| THE 2D SYSTOLIC ARRAY ARCHITECTURE |
+-------------------------------------------------------------------------------+
Inputs Stream In from Left --->
v
+------+ +------+ +------+
| Cell | ----> | Cell | ----> | Cell | Weights are pre-loaded and stationary.
+------+ +------+ +------+
| | |
v v v
+------+ +------+ +------+
| Cell | ----> | Cell | ----> | Cell | Data flows directly to neighbor cells
+------+ +------+ +------+ without touching a central register file!
| | |
v v v
Accumulated Outputs Flow Downward to Memory
How a Systolic Array Works
The word systolic comes from medical biology: mimicking the rhythmic pumping of blood through the human heart.
- Stationary Weights: Matrix weights are pre-loaded directly into local registers inside a 2D grid of $128 imes 128$ or $256 imes 256$ multiply-accumulate cells.
- Pulsing Dataflow: Activation vectors stream into the array from the left. On each clock cycle, a cell multiplies its pre-loaded weight by the incoming input, adds the accumulated sum from the cell above it, and passes the result directly to its neighbor cell to the right and below.
- Zero Register Read/Write Latency: Data flows directly through silicon wires from transistor to transistor. Intermediate results never touch a register file or main memory bus until the full matrix product emerges from the bottom of the grid.
This eliminates register file energy consumption and allows TPUs to achieve unmatched compute density for transformer training and inference.
Silicon Architecture Comparison Matrix
| Dimension | CPU | GPU (e.g., H100) | TPU (e.g., TPU v5p) |
|---|---|---|---|
| Core Philosophy | Minimize sequential latency | Maximize parallel throughput | Hardwired matrix streaming |
| Silicon Allocation | 70% Cache & Control Logic | 80% Arithmetic ALUs | Massive 2D Systolic Arrays |
| Execution Pattern | Out-of-order, speculative | SIMT (32-thread lockstep Warps) | Dataflow systolic pulse |
| Branching Efficiency | Maximum (Branch predictor) | Poor (Warp divergence penalty) | Extremely poor (No branches) |
| Optimal Precision | FP32 / FP64 | FP16 / BF16 / FP8 | Bfloat16 / Int8 / FP8 |
| Ideal Workloads | Operating systems, SQL DBs | General AI training & graphics | Large-scale LLM matrix training |
Summary and Key Takeaways
- CPUs optimize for control flow latency: CPUs dedicate physical silicon to branch prediction, out-of-order execution, and multi-level caching to accelerate unpredictable sequential threads.
- GPUs optimize for parallel arithmetic: GPUs deploy thousands of simple ALUs executing uniform instructions across 32-thread warps, excelling at parallel array math.
- Avoid warp divergence on GPUs: Branching code inside GPU kernels forces serial execution; keep GPU workloads uniform and branchless.
- TPU systolic arrays eradicate memory bus stalls: By streaming data directly between neighbor arithmetic cells, systolic arrays eliminate intermediate register reads and writes.
- Match chip architecture to mathematical structure: Run operating systems and databases on CPUs, diverse parallel computing on GPUs, and uniform hyper-scale matrix multiplications on TPUs.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.