Think of a CPU as a 4-lane highway. Each lane is wide, fast, and can handle any vehicle — sports cars, trucks, buses. The lanes have traffic lights that adapt in real time, ramps that predict where cars are going, and emergency shoulders for unexpected situations.

A modern CPU like the Intel Core i9-14900K has 24 cores running at up to 6.0 GHz. Each core has branch prediction, out-of-order execution, speculative execution, and deep cache hierarchies. These features make each core incredibly smart — it can handle complex, unpredictable tasks with minimal wasted cycles.

This design is perfect for tasks like running your operating system, compiling code, or serving web requests — workloads where each step depends on the result of the previous one.

When you send a prompt to an LLM, here’s what actually happens at the hardware level:

1. Token embedding: Your words become vectors (arrays of numbers). Each token is a vector of 4,096–12,288 floating-point numbers.

2. Attention layers: Each layer multiplies your token vectors by three huge weight matrices (Q, K, V) — each one is [hidden_dim × hidden_dim]. For a 70B model, that’s [8192 × 8192] matrices.

3. Feed-forward layers: Another pair of matrix multiplications, typically [hidden_dim × 4×hidden_dim].

4. Repeat: A 70B model has ~80 layers. Each layer does 4–6 matrix multiplications. That’s 320–480 matrix multiplications per token.

If a CPU is a 4-lane highway, a GPU is a 10,000-lane road. Each lane is narrow — it can only handle bicycles, not trucks. But when your job is to move 10,000 identical packages from point A to point B, 10,000 bicycles beat 4 trucks every time.

An NVIDIA H100 GPU has 16,896 CUDA cores and 528 Tensor Cores. Each CUDA core is simple — no branch prediction, no speculative execution, no out-of-order logic. It does one thing: take two numbers, multiply them, add to an accumulator.

The magic is that all 16,896 cores do this simultaneously. In a single clock cycle, the H100 performs thousands of multiply-add operations in parallel. This is the SIMT (Single Instruction, Multiple Threads) execution model.

Researchers benchmarked a 4096×4096 matrix multiplication — the kind of operation that happens hundreds of times per token in an LLM:

Sequential CPU (single core): The baseline. One core multiplying row by column, element by element. Painfully slow.

Parallel CPU (all cores): Using OpenMP to spread the work across all CPU cores gives a 12–14x speedup. Significant, but still limited by core count.

GPU (CUDA): The same matrix multiply on a GPU achieves a 593x speedup over sequential CPU and 45x over optimized parallel CPU.

For larger matrices (8192×8192, 16384×16384), the GPU advantage grows even further because GPUs scale better with problem size.

Imagine a kitchen with 100 chefs (GPU cores), but only one narrow door to the pantry (memory bus). The chefs can cook incredibly fast, but they spend most of their time waiting for ingredients.

This is the memory wall — the gap between how fast processors can compute and how fast memory can deliver data. It’s the single biggest bottleneck in AI hardware.

CPU memory bandwidth: A server CPU gets ~90–460 GB/s from DDR5 RAM. That’s the equivalent of a single-lane road to the pantry.

GPU memory bandwidth: An H100 gets 3,350 GB/s from HBM3 memory. That’s a 37-lane highway to the pantry. The newer B200 pushes this to 8,000 GB/s.

Let’s do some back-of-envelope math to understand why CPUs simply cannot train modern AI models:

Training Llama 3 70B required approximately 6.4 million GPU-hours on H100 GPUs. Meta used 16,384 H100 GPUs running for about 54 days.

Each H100 delivers ~990 TFLOPS at FP16 with Tensor Cores. A top server CPU delivers ~5 TFLOPS at FP32 (and less at FP16 since CPUs lack native FP16 acceleration).

Even being generous and assuming the CPU could match its FP32 rate at FP16, you’d need ~200x more time on CPUs. That 54-day training run becomes ~29 years on the same number of CPUs. And you’d need 16,384 of the most expensive server CPUs on the planet.

Training is like building a factory — massive upfront investment, enormous compute, done once (or a few times). GPUs are absolutely essential here. No debate.

Inference is like running the factory — serving predictions to users. Here, the picture is more nuanced:

For large models (70B+), GPUs are still essential for inference. The model weights alone need 140+ GB of memory, and you need high bandwidth to generate tokens fast.

For small models (1B–7B), CPUs can actually work. A quantized 7B model runs at 10–30 tokens/sec on a modern CPU. Not fast, but usable for batch processing. For models under 1.5B parameters, CPUs can even outperform budget GPUs by 1.3x.

The CPU-GPU performance gap doesn’t just affect training speed. It cascades through every infrastructure decision:

GPU scarcity → GPU pricing: H100s cost $25,000–$40,000 each. Demand far exceeds supply. This drives the entire cloud GPU market.

GPU power → cooling crisis: An H100 draws 700W. A B200 draws 1,000W. A rack of 8 GPUs needs 8,000W just for compute. This is why data centers are moving to liquid cooling.

GPU memory limits → distributed training: A 70B model doesn’t fit on one GPU. You need 4–8 GPUs connected by fast interconnects. This drives NVLink, InfiniBand, and cluster networking.

GPU efficiency → cost optimization: At $3–4/hr per GPU, keeping GPUs idle is burning money. This drives scheduling, orchestration, and utilization optimization.