Renting (API) — Pay per use, no maintenance, move anytime. Like renting an apartment: you pay monthly, the landlord handles plumbing and repairs. Buying (Self-hosting) — Purchase GPUs, run your own models. Like buying a house: big upfront cost, you handle maintenance, but no monthly rent. Building (Custom infra) — Design custom hardware and software stacks. Like building a house from the ground up: maximum control, maximum complexity.

Most teams should rent (use APIs). APIs win for 87% of use cases because they eliminate infrastructure complexity, scale instantly, and require zero GPU expertise. Self-hosting only makes sense at extreme volume (>10B tokens/month) or with strict regulatory requirements (HIPAA, data sovereignty). Building custom infrastructure is reserved for hyperscalers and AI labs.

GPU memory (HBM) determines how large a model can fit on a single chip. A 70B parameter model needs ~140GB in FP16, requiring 2 H100s but only 1 H200. Fewer GPUs = lower cost, simpler infrastructure, and better utilization. The memory race is as important as the compute race.

Each generation delivers 2–3x inference throughput over the previous. B200 processes roughly 2.5x more tokens per second than H100 for the same model. This is a key driver of the cost collapse from Chapter 1 — newer hardware makes each token cheaper to serve, which providers pass on (partially) as lower API prices.

NVIDIA operates at 79–89% gross margins on datacenter GPUs. A B200 costs ~$6,400 to manufacture and sells for $40,000–$50,000. The biggest cost driver is HBM memory, which accounts for 35–47% of manufacturing cost ($2,900 for B200’s HBM3e). Advanced packaging (CoWoS-L) adds another $1,100.

The hourly GPU rate is just the start. Egress fees (data transfer out) can add 20–30% for high-throughput inference. Storage premiums for model weights and data. Virtualization overhead reduces effective GPU performance by 5–15%. Networking for multi-GPU setups. Total hidden costs typically add 20–40% on top of the base GPU rate.

APIs win until you hit roughly 10 billion tokens/month (~500M tokens/day). Below that volume, the operational overhead of self-hosting exceeds the API cost savings. Above that volume, the per-token economics of dedicated GPUs start to win — but only if you can maintain >60% GPU utilization and have the engineering team to support it.

HIPAA, SOC 2, GDPR, data sovereignty. When regulations prohibit sending data to third-party APIs, self-hosting is the only option. Healthcare, finance, and government often require on-premises or private-cloud inference. The cost premium is a compliance cost, not a technology choice.

>10B tokens/month with consistent, predictable demand. At this scale, dedicated GPUs at high utilization beat per-token API pricing. But the demand must be consistent — bursty workloads still favor APIs because idle GPUs cost the same as busy ones.

Sub-100ms inference. API calls include network latency, queue time, and shared infrastructure overhead. Self-hosted models on dedicated GPUs can achieve 2–5x lower latency for latency-critical applications like real-time trading, gaming, or interactive voice.

Fine-tuned or proprietary models that can’t run on API providers. If you’ve trained a custom model on proprietary data, you need your own inference infrastructure. This is increasingly common for domain-specific applications in legal, medical, and scientific fields.

A GPU costs the same whether it’s processing tokens or sitting idle. Most self-hosted deployments run at 15–30% average utilization because demand is bursty — peak hours see high load, but nights and weekends are quiet. At 20% utilization, your effective cost per token is 5x higher than the theoretical maximum.

Techniques to push utilization higher: Request batching (group multiple requests into single GPU passes). Multi-model serving (run different models on the same GPU based on demand). Spot/preemptible instances (use cheap GPU time for non-urgent batch work). Auto-scaling (spin GPUs up/down with demand, though this has cold-start penalties).

Chapter 6 covers the optimization playbook — the water conservation analogy. Whether you use APIs or self-host, these techniques (caching, routing, compression, distillation, batching) can cut your bill by 60–70%. They work at every scale and every infrastructure choice.

APIs win for 87% of use cases. NVIDIA GPUs carry 79–89% margins, with B200 costing $6,400 to make and selling for $40K+. Cloud GPU pricing varies 3x between specialized and hyperscaler providers. Self-hosting hidden costs are 3–5x raw GPU price. The break-even is ~10B tokens/month at >60% utilization. Four scenarios justify self-hosting: regulated data, ultra-high volume, latency requirements, and custom models.