Training processes massive batches of data in a predictable loop — forward pass, backward pass, gradient update. The GPU stays busy with large matrix multiplications. Inference is the opposite: unpredictable request arrivals, variable sequence lengths, and a two-phase execution pattern that fundamentally changes the bottleneck.

Prefill phase: Process the entire input prompt at once. This is compute-bound — similar to training. A 4,096-token prompt on an H100 takes ~50ms.

Decode phase: Generate tokens one at a time, each requiring a full attention pass over all previous tokens. This is memory-bandwidth-bound because you’re reading the entire KV cache for each new token but only producing a single output.

The result: during decoding, an H100’s 3,958 TFLOPS (FP8) sits mostly idle while the 3.35 TB/s memory bus becomes the bottleneck. A single request might use only 1–5% of the GPU’s compute capacity during decode.

In transformer attention, each token needs to attend to all previous tokens. Without caching, generating token N requires recomputing attention for all N-1 previous tokens — O(N²) total work for a sequence. The KV cache stores the Key and Value projections from previous tokens so each new token only computes its own K/V and attends to the cached values.

This trades compute for memory. Instead of recomputing, you store and read. For large models with long contexts, this cache becomes enormous.

An H100 has 80 GB of HBM3. For Llama 3 70B in FP16, the model weights alone consume ~140 GB (needs 2 GPUs via tensor parallelism). On each GPU, roughly 10 GB is left for KV cache after weights and activations. That’s enough for only ~8 concurrent users at 4K context.

With FP8 quantization, weights shrink to ~70 GB (1 GPU), leaving ~8 GB for cache. Still only ~6 users. The KV cache is the primary factor limiting how many users a single GPU can serve simultaneously.

Long-context models make this worse: A 128K-context request consumes 40.96 GB of cache — more than the model weights. A single long-context user can monopolize an entire GPU.

Before PagedAttention, inference engines pre-allocated contiguous memory blocks for each request’s KV cache based on the maximum possible sequence length. If a model supports 4,096 tokens but a request only uses 500, the remaining 3,596 tokens worth of memory sits wasted — reserved but empty.

Across many concurrent requests, this wastes 60–80% of GPU memory. It’s like reserving an entire row of seats at a movie theater for each person, even though most people come alone.

Additionally, contiguous allocation causes external fragmentation. Even if total free memory is sufficient, it may be scattered in non-contiguous chunks too small for any single request.

PagedAttention (introduced by vLLM in 2023) borrows the concept of virtual memory paging from operating systems:

1. Fixed-size blocks: KV cache is divided into pages of 16 tokens each (configurable). Each page is a small, fixed-size memory block.

2. Block table: A mapping table (like a page table in OS virtual memory) tracks which physical memory blocks belong to which sequence. Blocks don’t need to be contiguous.

3. On-demand allocation: New blocks are allocated only when a sequence actually generates tokens that need them. No pre-allocation of maximum length.

4. Copy-on-write: When multiple requests share a common prefix (e.g., system prompt), they can share the same physical KV cache blocks until they diverge.

The simplest approach: collect N requests, process them together, wait for all to finish, then collect the next batch.

Problem: If one request generates 500 tokens and another generates 10, the short request’s GPU slot sits idle for 490 decode steps. With variable-length outputs (common in chat), this wastes 50–70% of potential throughput.

It’s like a bus that won’t leave until every passenger reaches their final destination — even if most people want to get off at the first stop.

Continuous batching (also called iteration-level scheduling) checks for completed sequences after every decode step. When a sequence finishes, its slot is immediately filled with a new request from the queue.

Prefill scheduling: New requests can be prefilled (prompt processing) while existing requests are decoding. The prefill is chunked into smaller pieces to avoid stalling decode latency for in-flight requests.

Result: GPU utilization jumps from 30–40% (static) to 70–90% (continuous). Throughput increases 2–3× with the same hardware.

A complementary technique: use a small “draft” model (e.g., 1B params) to generate K candidate tokens quickly, then verify all K in a single forward pass of the large model. If the draft model guesses correctly (common for predictable text), you generate K tokens for the cost of ~1 large-model step.

Typical speedup: 1.5–2.5× for code generation and structured text. Less effective for creative writing where the draft model’s predictions diverge.

Four engines dominate production LLM serving, accounting for ~85% of deployments:

vLLM — The flexibility champion. Open-source, hardware-agnostic (NVIDIA, AMD, TPU). PagedAttention, continuous batching, speculative decoding, LoRA hot-swapping. vLLM V1 (Jan 2025) redesigned the core for near-zero CPU overhead. Best for high-concurrency (50–100+ requests) and multi-modal workloads.

TensorRT-LLM — NVIDIA’s hardware-optimized engine. Aggressive kernel fusion, FP8/FP4 quantization, pipeline parallelism. 35–50% higher throughput than vLLM on identical NVIDIA hardware for single-model serving. Trade-off: 15–45 minute model compilation step and NVIDIA lock-in.

SGLang — Structured output specialist. RadixAttention for prefix caching, JSON schema enforcement, multi-LoRA batching. Best for DeepSeek models and workloads with high prompt overlap (RAG, chatbots).

Hugging Face TGI — Simplicity champion. Single Docker image, minimal config. Lower raw performance but fastest time-to-production.

Inference quantization is more aggressive than training because you don’t need gradients:

FP8 (W8A8): Weights and activations in 8-bit. Near-zero accuracy loss on most models. Halves memory vs FP16, doubles throughput. The default for production H100/H200 deployments.

INT4/GPTQ/AWQ (W4A16): 4-bit weights, 16-bit activations. Cuts weight memory by 4×. Llama 70B fits on a single GPU (17.5 GB weights). Small accuracy loss (~1–2% on benchmarks).

FP4 (Blackwell): NVIDIA B200 supports native FP4. Llama 70B weights in ~8.75 GB. Enables serving 70B models on a single GPU with room for large KV caches.

Many requests share common prefixes: system prompts, few-shot examples, RAG context. Prefix caching computes and stores the KV cache for these shared prefixes once, then reuses it across requests.

Impact: For a RAG application with a 2,000-token system prompt, prefix caching eliminates ~50ms of prefill latency per request and saves ~8 MB of KV cache memory per concurrent user. At 100 concurrent users, that’s 800 MB saved and 5 seconds of GPU time saved per second.

Prefill is compute-bound; decode is memory-bound. Running both on the same GPU means neither phase gets optimal hardware:

Disaggregated serving splits prefill and decode onto separate GPU pools. Prefill GPUs are configured for maximum compute (higher clock, less memory). Decode GPUs are configured for maximum memory bandwidth and capacity.

Benefit: 20–40% throughput improvement at the system level. The trade-off is increased complexity and network overhead for transferring KV cache between GPU pools.

Most organizations don’t have enough traffic for a single model to saturate a GPU 24/7. A customer-facing chatbot might peak at 100 requests/second during business hours but drop to 5 requests/second at night. Dedicating 8× H100s ($250K+/year) to a model that’s idle 60% of the time is wasteful.

Multi-tenancy solves this by sharing GPU resources across multiple models, users, or workloads. The challenge: GPU memory isn’t easily shared, model loading takes 30–120 seconds, and latency SLAs vary by customer.

1. Time-slicing: Load/unload models based on demand. Works for infrequent models but model loading latency (30–120s) makes it impractical for interactive workloads.

2. MPS (Multi-Process Service): NVIDIA’s GPU sharing at the CUDA level. Multiple processes share a GPU with isolated memory spaces. Good for small models but no memory overcommit.

3. MIG (Multi-Instance GPU): Hardware-level GPU partitioning on A100/H100. Splits one GPU into up to 7 isolated instances. Each instance has guaranteed memory and compute. Best for hard isolation requirements.

4. LoRA multiplexing: Load one base model + many LoRA adapters (1–5% of base model size). Serve different fine-tuned variants from a single GPU. vLLM supports hot-swapping LoRA adapters with zero downtime.

Scale-to-zero: Unload models with no traffic. First request triggers loading (cold start: 30–120s). Acceptable for internal/batch workloads, not for customer-facing APIs.

Predictive scaling: Use historical traffic patterns to pre-scale. If traffic peaks at 9 AM, start scaling at 8:45 AM. Reduces cold starts by 80–90%.

Request-based autoscaling: Scale replicas based on queue depth or P99 latency. Target: keep P99 latency below SLA (e.g., 500ms TTFT). Kubernetes HPA with custom metrics from the inference engine.

The fundamental unit of inference cost is dollars per million tokens. This depends on three factors: GPU cost per hour, tokens generated per hour, and overhead (networking, storage, engineering).

For a well-optimized vLLM deployment of Llama 3 70B on 4× H100 (FP8):

Self-hosting breaks even when your monthly token volume exceeds the point where GPU costs (amortized) are less than API costs. The crossover depends on utilization:

Engineering: 0.5–2 FTEs to manage inference infrastructure ($100–400K/yr).
Monitoring: Prometheus, Grafana, custom dashboards for latency/throughput SLAs.
Model updates: New model versions require re-quantization, re-benchmarking, A/B testing.
Redundancy: Need 2× capacity for zero-downtime deployments during updates.
Edge cases: OOM errors, CUDA crashes, driver updates, security patches.