AI customization is not a binary choice. It’s a spectrum with five distinct levels, each with different costs, timelines, and capabilities:

Level 1: API + Prompting — Use a foundation model via API with well-crafted prompts. Zero setup cost. Sufficient for 60–70% of enterprise use cases.

Level 2: RAG (Retrieval-Augmented Generation) — Ground the model in your proprietary data without changing the model itself. $0.01–$0.05 per query.

Level 3: Fine-tuning (LoRA/PEFT) — Adjust the model’s behavior on your specific tasks. $500–$2,000 for a 70B model.

Level 4: Continued pre-training — Teach the model an entirely new domain (medical, legal, financial). $10,000+. Requires substantial domain-specific data.

Level 5: Training from scratch — Build a foundation model from the ground up. $10M–$100M+. Only justified for organizations with unique data at massive scale and deep ML expertise.

You send requests to a model hosted by a provider (OpenAI, Anthropic, Google) and pay per token. No infrastructure to manage, no ML expertise required. The model is shared across all customers, but your data is processed and returned — not used for training (under enterprise agreements). You customize behavior entirely through system prompts, few-shot examples, and output format constraints (Chapter 16).

GPT-4o — ~$4–5 per million tokens.
Claude Sonnet — ~$3 per million input tokens.
Gemini Flash — ~$0.075 per million tokens (the cost leader).

For context: 1 million tokens is roughly 750,000 words — about 10 full-length novels. At Gemini Flash pricing, processing 10 novels costs less than a cup of coffee.

General-purpose tasks — Summarization, translation, drafting, Q&A, analysis.
Variable workloads — You pay only for what you use. No idle infrastructure.
Rapid experimentation — Switch between models in minutes, not months.
Multi-model strategies — Route different tasks to different models based on complexity and cost.

Fine-tuning takes a pre-trained foundation model and adjusts its weights using your specific data. The model retains its broad knowledge but develops specialized expertise in your domain, terminology, and output style. Modern fine-tuning uses LoRA (Low-Rank Adaptation) — a technique that modifies only a small fraction of the model’s parameters (typically 0.1–1%), making it dramatically cheaper and faster than full fine-tuning.

LoRA fine-tuning (7B model) — $50–$500 one-time.
LoRA fine-tuning (70B model) — $500–$2,000 one-time.
Full fine-tuning — $5,000–$50,000.
Enterprise-scale fine-tuning — $12,000–$180,000.

These costs are amortized across millions of inference calls. The per-request cost drops because fine-tuned models need 50–90% shorter prompts — the knowledge is baked into the model, not repeated in every prompt.

Output consistency — 98–99.5% format adherence vs. 85–95% with prompting alone.
Domain terminology — Medical, legal, or technical vocabulary used correctly and naturally.
Behavioral patterns — Specific tone, decision-making style, or response structure.
Cost reduction at scale — Processing 10,000 documents daily: ~$50K/year via API vs. ~$5K with fine-tuned on-premises model. A 90% cost reduction.

Knowledge distillation uses a large, expensive model (the “teacher”) to generate training data for a small, cheap model (the “student”). You run your use case through GPT-4 or Claude to produce thousands of high-quality examples, then fine-tune a small open-source model (like Phi-3.5 or LLaMA 8B) on those examples. The student model learns to mimic the teacher’s behavior on your specific task at a fraction of the cost.

A distilled 8B model typically achieves 90–95% of the quality of the 70B teacher at a 25× reduction in inference cost. For high-volume production tasks where the quality bar is “good enough, consistently,” this is transformative. A task that costs $10 per 1M tokens via GPT-4 Turbo costs $0.40 with a distilled model on modest hardware.

High-volume, well-defined tasks — Classification, extraction, formatting, routing. Tasks where the output space is bounded and predictable.

Latency-sensitive applications — Small models run in 50–80ms vs. 400ms+ for cloud APIs. For real-time applications (chatbots, search, fraud detection), this latency difference is significant.

Data-sensitive environments — The distilled model runs on your infrastructure. No data leaves your network.

Fine-tuned models excel within their training distribution but experience complete accuracy collapse outside it. Air Canada’s chatbot invented refund policies that didn’t exist. Amazon’s Rufus product assistant matched only 32% of products accurately when queries deviated from training patterns. The model becomes a specialist that is confidently wrong when asked anything outside its narrow expertise.

Fine-tuning amplifies whatever is in your training data — including errors, biases, and inconsistencies. You need at minimum 1,000 high-quality examples, and ideally 5,000–10,000 for production-grade results. Curating this data is often the most expensive and time-consuming part of the process, yet it’s frequently underestimated in project planning.

Model drift — As your business changes, the fine-tuned model becomes stale. You need a retraining pipeline, not a one-time project.
Evaluation infrastructure — You need automated tests to detect regression when you retrain.
Version management — Multiple model versions in production, with rollback capability.
Talent requirements — Fine-tuning requires ML engineering expertise that commands $200K–$400K salaries.

Models like Meta’s LLaMA, Mistral, and DeepSeek can be downloaded and run on your own infrastructure. No data leaves your network. No per-token fees. No dependency on a third-party provider. You have full control over the model, its behavior, and its availability. The trade-off: you own the infrastructure, the operations, and the expertise required to run it.

Entry-level (7B model) — A single A10G GPU (~$1,500/month cloud). Handles moderate traffic.
Production (70B model) — Multiple A100 GPUs (~$33,000/month cloud). Enterprise-scale throughput.
High-performance cluster — 8× A100 instance (~$32,770/month). Required for large models or high concurrency.

These costs are fixed regardless of usage, making self-hosting economical only at high volume (10M+ tokens/day).

Regulated industries — Healthcare, defense, financial services where data cannot leave your infrastructure under any circumstances.
High-volume production — At 50M tokens/day, self-hosted costs $2.20/M tokens vs. $10.00 for GPT-4 Turbo API.
Latency requirements — On-premises models eliminate network round-trips (50–80ms vs. 400ms+).
Customization needs — Full control over fine-tuning, quantization, and serving configuration.

Step 1: Start with API + prompting. Test your use case with a foundation model and well-crafted prompts. If quality meets your bar, stop here. This is the right answer for most use cases.

Step 2: Add RAG for proprietary data. If the model needs access to your documents, knowledge base, or real-time data, add retrieval (Chapter 17). This keeps the model current without retraining.

Step 3: Layer prompt caching. For static system prompts and policies, caching reduces costs by ~90% after the first call.

Step 4: Fine-tune only if Steps 1–3 fall short. Fine-tune when you need >95% output consistency, domain-specific behavior at scale, or significant cost reduction on high-volume tasks.

Step 5: Distill for production scale. Once a fine-tuned large model validates the use case, distill to a smaller model for cost-effective deployment.

Step 6: Self-host only with clear justification. Data sovereignty requirements, extreme volume (>10M tokens/day), or latency constraints that APIs cannot meet.

Volume <100K requests/month? → API + prompting. Don’t over-invest.

Need proprietary data access? → Add RAG. Keep the model general, make the data specific.

Output consistency <95%? → Consider fine-tuning. The 85% → 99% consistency jump justifies the investment for production systems.

Volume >100K requests/month on a validated task? → Fine-tune + distill. The cost savings compound at scale.

Data cannot leave your network? → Self-host open-source models. Accept the operational overhead.

Unique domain with no public training data? → Continued pre-training. Rare but necessary for specialized fields.

Tasks change frequently? → Stay with APIs. Fine-tuned models are rigid; prompts are flexible. Don’t fine-tune a moving target.