Think of a GPU as a massive industrial park with dozens of identical factories. Each factory is a Streaming Multiprocessor (SM) — a self-contained processing unit with its own workers, tools, and local storage.

The NVIDIA A100 has 108 SMs. The H100 has 132 SMs. The B200 has 160 SMs. Each SM operates independently, running its own set of threads on its own data.

Inside each SM, you’ll find:
• CUDA cores (the general workers)
• Tensor Cores (the specialized matrix machines)
• Warp schedulers (the foremen who assign work)
• Register file + shared memory (the local stockroom)

The GPU’s job is to keep all these factories running at full capacity simultaneously.

A CUDA core is like a worker with a single screwdriver. They can do exactly one thing per clock cycle: take two numbers, multiply them (or add them), and produce a result.

No branch prediction. No speculative execution. No out-of-order logic. Just: multiply, add, done. Next.

This simplicity is the point. Because each core is so simple, you can fit thousands of them on a single chip. The H100 has 16,896 CUDA cores, each running at ~1.98 GHz.

Each CUDA core handles either a floating-point (FP32, FP64) or integer (INT32) operation. Modern architectures can execute FP32 and INT32 operations simultaneously on separate datapaths within the same SM, effectively doubling throughput for mixed workloads.

If a CUDA core is a worker with a screwdriver (one screw at a time), a Tensor Core is a worker with a power drill that drives 64 screws simultaneously.

A Tensor Core performs an entire 4×4 matrix multiply-and-accumulate in a single instruction. That’s 64 multiply-add operations at once — roughly 8x the throughput of CUDA cores for matrix work.

Introduced in NVIDIA’s Volta architecture (2017), Tensor Cores have evolved dramatically:

• Volta (2017): FP16 input, FP32 accumulate
• Ampere (2020): Added BF16, TF32, INT8, INT4
• Hopper (2022): Added FP8, larger matrix sizes
• Blackwell (2024): Added FP4, 2nd-gen Transformer Engine

Imagine a factory where workers operate in teams of 32. Every team member does the exact same task, but on different pieces of material. If one team member needs to wait for a delivery (memory fetch), the foreman instantly switches to a different team that’s ready to work. No idle time.

This is how GPU warps work. A warp is a group of 32 threads that execute the same instruction simultaneously (SIMT model). Each SM has 4 warp schedulers that can issue instructions to different warps every cycle.

The magic trick: when a warp stalls waiting for memory (which takes 200–400 cycles), the scheduler instantly switches to another ready warp. With enough warps in flight, the GPU hides memory latency entirely — there’s always a ready warp to execute.

Think of GPU memory like a series of storage locations, each farther from the worker but larger:

Registers (your pockets): Fastest access, ~0 cycles. Each thread has its own registers. The H100 has 256 KB of register file per SM — that’s 33 MB total across all SMs.

Shared Memory (your desk): ~20 cycles. Shared among all threads in a block. 228 KB per SM on H100. Used for data that multiple threads need to access.

L2 Cache (the filing cabinet): ~200 cycles. 50 MB on H100. Shared across all SMs. Caches frequently accessed data from HBM.

HBM (the warehouse): ~300–400 cycles. 80 GB on H100. This is where model weights and activations live. High capacity but relatively slow to access.

Precision formats are like photo resolution. A 4K photo (FP32) captures every detail but takes lots of storage and bandwidth. A 1080p photo (FP16) looks almost identical to the human eye but uses half the space. A thumbnail (FP8) is fine for previewing but you wouldn’t print it.

FP32 (32 bits): Full precision. 1 sign + 8 exponent + 23 mantissa bits. The gold standard, but slow and memory-hungry.

BF16 (16 bits): Google’s brain float. Same exponent range as FP32 (so same number range) but fewer mantissa bits. Ideal for training — maintains dynamic range while halving memory.

FP16 (16 bits): IEEE half-precision. Smaller range than BF16 but more mantissa precision. Good for inference.

FP8 (8 bits): Introduced with Hopper. Two variants: E4M3 (more precision) and E5M2 (more range). Doubles throughput again with minimal quality loss for inference.

Let’s trace a single matrix multiplication (the core of every attention layer) through the GPU:

1. Load from HBM: Weight matrix tiles and input activations are loaded from HBM3 into L2 cache, then into shared memory. This is the slowest step (~300 cycles).

2. Tile into shared memory: The matrices are split into small tiles (e.g., 128×128) that fit in shared memory. Each SM works on one tile.

3. Feed Tensor Cores: Data moves from shared memory into registers, then into Tensor Cores. Each Tensor Core computes a 4×4 (or larger) matrix multiply-accumulate.

4. Accumulate results: Partial results are accumulated in registers (FP32 precision for accuracy, even if inputs are FP16).

5. Write back: Final results go from registers → shared memory → L2 → HBM for the next layer to consume.

NVIDIA releases a new GPU architecture roughly every two years. Each generation brings major improvements:

Volta (2017, V100): Introduced Tensor Cores. The GPU that made modern deep learning training practical. 125 TFLOPS FP16.

Ampere (2020, A100): Added BF16 and TF32 support. Structural sparsity for 2x inference boost. 312 TFLOPS FP16. The workhorse of the GPT-3 era.

Hopper (2022, H100): Added FP8 and the Transformer Engine that automatically manages precision. 990 TFLOPS FP16. The GPU that trained GPT-4 and Llama 3.

Blackwell (2024, B200): Dual-die chiplet design with 208 billion transistors. Added FP4. 2nd-gen Transformer Engine. ~2,250 TFLOPS FP16. The current flagship.