---
topic: ai-technology
author: Crashtech Editorial
published: Nov 15, 2026 · read: 3 min
---

Why AI Needs GPUs and TPUs: Matrix Multiplication, Tensor Cores, and the Memory Wall

Why CPUs fail on deep learning: the physics of SIMT Tensor Cores, systolic arrays, High Bandwidth Memory, and breaking through the Von Neumann memory wall.

– –

When interacting with ChatGPT, Claude, or Gemini, it is easy to view artificial intelligence through a purely algorithmic lens: tokenizers, transformer blocks, prompt templates, and loss functions.

However, artificial intelligence is not merely a software discipline. At its core, AI is an uncompromising physics problem: the challenge of transferring billions of electrons through physical silicon and moving massive terabytes of data between memory storage and arithmetic processors under strict thermal and energy limits.

Modern large language models cannot be trained or served on traditional Central Processing Units (CPUs). Understanding why requires examining the fundamental mathematical structure of neural networks, the failure of the Von Neumann architecture, and the hardware breakthroughs of Graphics Processing Units (GPUs) and Tensor Processing Units (TPUs).

+--------------------------------------------------------------------------+
|                  The Fundamental Semiconductor Divide                    |
+--------------------------------------------------------------------------+
| Central Processing Unit (CPU) | Optimized for low-latency serial logic,  |
|                               | unpredictable branching, and deep caches |
| AI Accelerators (GPU / TPU)   | Optimized for massive parallel floating- |
|                               | point matrix arithmetic and HBM bandwidth|
+--------------------------------------------------------------------------+

The Mathematics: Neural Networks as Matrix Factories

At the silicon layer, deep learning is overwhelmingly a single mathematical operation repeated billions of times: Matrix Multiplication ($Y = W \cdot X + B$).

Input Vector (X)  x  Weight Matrix (W)  +  Bias Vector (B)  =  Layer Output (Y)

In a 70-billion parameter language model, a single forward pass requires over 140 trillion floating-point operations (FLOPs).

Critically, this workload possesses an extraordinary property: embarrassing parallelism. Computing row 1 multiplied by column 1 does not depend on row 2 multiplied by column 2. Trillions of individual multiply-accumulate operations can be executed concurrently with zero inter-thread communication.


Why CPUs Collapse: The Von Neumann Memory Wall

The CPU was engineered for general-purpose computing: running web browsers, operating system kernels, and database servers. Because general code is full of unpredictable if-else branches, CPUs dedicate the vast majority of their silicon area to control logic:

  • Branch Prediction Units: Predict which path conditional code will follow.
  • Out-of-Order Schedulers: Reorder instructions dynamically to keep execution pipelines full.
  • Deep L1/L2/L3 Cache Hierarchies: Attempt to hide memory access delays for random pointer dereferences.
+---------------------------------------------------------------+
|                      CPU Silicon Die Area                     |
+---------------------------------------------------------------+
| [Control Logic: Branch Predictors & Out-of-Order]  (45% Area) |
| [L1 / L2 / L3 SRAM Cache Hierarchies]              (40% Area) |
| [Arithmetic Logic Units (ALUs)]                    (Only 15%!)|
+---------------------------------------------------------------+

When running an LLM, branch prediction is useless: neural networks execute identical deterministic matrix transformations across every layer. Billions of CPU transistors sit completely idle.

The Memory Wall

Even if a CPU had thousands of arithmetic units, it would choke on the Memory Wall: the widening gap between processor cycle times and memory bus bandwidth.

A 70-billion parameter model in 16-bit precision occupies roughly 140 gigabytes of memory. To generate a single output token, the processor must read every single one of those 140 gigabytes from RAM into registers to perform matrix math. Standard server DDR5 memory delivers roughly 100 GB/s bandwidth. Reading 140 GB across a DDR5 bus takes over 1.4 seconds per token: unacceptably sluggish for interactive conversational products.


How GPUs Solve the Problem: SIMT and Tensor Cores

NVIDIA GPUs replaced complex CPU cores with thousands of simpler arithmetic units organized around Single Instruction, Multiple Threads (SIMT):

+---------------------------------------------------------------+
|                 NVIDIA H100 Architecture Scale                |
+---------------------------------------------------------------+
| CUDA Cores:           Nearly 17,000 parallel execution units  |
| Tensor Cores:         528 Fourth-Generation Matrix Engines    |
| Memory Bandwidth:     Over 3,350 GB/s (High Bandwidth Memory) |
+---------------------------------------------------------------+

Tensor Cores: 64 Operations in a Single Clock Cycle

Standard processor cores compute scalar arithmetic: one multiply or one add per cycle. Starting with the Volta architecture, NVIDIA introduced Tensor Cores: hardwired functional units that execute an entire $4 \times 4$ matrix multiply-accumulate in a single clock cycle:

[Tensor Core 4x4 Matrix Multiply-Accumulate]
Matrix A (4x4)  x  Matrix B (4x4)  +  Matrix C (4x4)  =  Matrix D (4x4)
Total Operations = 16 Multiplies + 16 Additions + 16 Accumulations = 64 Ops!

High Bandwidth Memory (HBM)

To feed these thousands of arithmetic cores, GPUs abandoned traditional motherboard DIMM slots. High Bandwidth Memory (HBM3) stacks DRAM dies vertically using microscopic through-silicon vias (TSVs). The stacks sit directly on a shared silicon interposer millimeters away from the GPU compute die:

[HBM3 Stack] <===(Microns-long Silicon Interposer)===> [GPU Compute Die]
Achieves 3,350 GB/s bandwidth (Streaming a 140GB model in 0.04 seconds!)

TPUs: Google’s Systolic Array Bet

While GPUs evolved from graphics processing heritage, Google designed the Tensor Processing Unit (TPU) from scratch as a pure matrix accelerator:

+-------------------------------------------------------------------------+
|                  Systolic Array Weight-Stationary Dataflow              |
+-------------------------------------------------------------------------+
| Weights loaded once into 256x256 grid cells (65,536 MAC units).         |
| Inputs stream horizontally from the left.                                |
| Accumulated products flow vertically downward.                           |
| Intermediate results pass directly between neighbors without hitting RAM!|
+-------------------------------------------------------------------------+

Reading data from DRAM consumes 200 times more electrical energy than executing a floating-point multiplication. By holding weights stationary inside on-chip registers and passing data directly between physical adjacent cells, the TPU eliminates memory bus traffic, achieving dramatic power efficiency gains over general-purpose silicon.


Hardware Defines AI Progress

The transition from CPUs to GPUs and TPUs marks a historic turning point in computer engineering: moving away from serial von Neumann logic toward high-bandwidth, massively parallel linear algebra engines. In modern artificial intelligence, models are shaped by the physical realities of memory bandwidth and silicon interconnects.

Advertisement

Frequently asked questions

What is the Memory Wall and why does it dictate LLM inference speed?

The Memory Wall is the severe throughput gap between fast processor arithmetic and slow memory bus bandwidth. To generate a single token, a 70B model must stream all 140GB of weights from memory into registers, making memory bandwidth the primary bottleneck.

How do Tensor Cores differ from standard CPU and GPU arithmetic logic units?

Standard ALUs execute scalar operations on individual numbers one by one. A Tensor Core is a hardwired matrix accelerator that executes an entire 4x4 matrix multiply-accumulate operation (64 arithmetic operations) in a single clock cycle.

Why is High Bandwidth Memory (HBM) critical for modern AI accelerators?

HBM stacks DRAM memory dies vertically using microscopic through-silicon vias (TSVs) placed on a shared silicon interposer directly adjacent to the GPU die, achieving over 3,350 GB/s bandwidth compared to ~100 GB/s on traditional DDR memory.

Why do CPUs devote most of their silicon die to non-arithmetic circuits?

CPUs are general-purpose processors designed for unpredictable code. They dedicate over 70% of their silicon to branch predictors, out-of-order execution schedulers, and deep cache hierarchies that sit completely idle during deterministic matrix multiplication.

Sources & further reading

/* Comments */