You can reduce your water bill three ways: (1) Use less — shorter showers, fix leaks (prompt compression, output length control). (2) Reuse — collect rainwater, recycle greywater (prompt caching, semantic caching). (3) Switch to a cheaper source — use well water instead of city water (model routing, distillation). AI cost optimization works the same way.

Model routing sends each request to the cheapest model that can handle it well. A classifier (itself a cheap model) evaluates incoming requests and routes simple tasks to budget models and complex tasks to premium models. Research shows 62% of agent tasks can route to budget models with zero quality loss.

The simplest router is a keyword/heuristic classifier that checks request length, presence of code blocks, or explicit complexity markers. More sophisticated routers use a small LLM (GPT-4o mini at $0.15/M) to classify the request before routing. The router cost is negligible compared to the savings from routing 60%+ of traffic to budget models.

When your requests share the same prefix (system prompt, tool definitions, few-shot examples), the provider can cache the processed KV-cache and reuse it across requests. Instead of reprocessing 2,000 tokens of system prompt for every request, the cached version is loaded instantly. Anthropic offers ~90% discount on cached reads. OpenAI offers ~50% discount.

Prompt caching also improves time-to-first-token by 13–31% because the model skips the prefill phase for cached tokens. You get both cost savings and faster responses — a rare win-win in optimization.

Semantic caching goes further: it caches entire responses for similar queries. When a new query is semantically similar to a previously answered one (measured by embedding cosine similarity), the cached response is returned without making an LLM call at all. This captures the 35–45% of queries that are repeated or near-identical in most production systems.

Remove redundancy from prompts. Most system prompts contain verbose instructions that can be condensed 30–50% without quality loss. Remove filler phrases, consolidate overlapping instructions, and use structured formats instead of prose. A 2,000-token system prompt compressed to 1,200 tokens saves 40% on that prefix across every request.

Since output tokens cost 3–8x more than input, controlling output length has outsized impact. Use max_tokens to cap responses. Add explicit length instructions: “Respond in 2–3 sentences” instead of “Explain thoroughly.” For structured output, use JSON mode to prevent verbose prose.

For multi-turn conversations, implement sliding window + summarization. Keep the last 3–5 turns in full, summarize older turns, and drop irrelevant history. This prevents the context from growing unboundedly and triggering the quadratic scaling costs from Chapter 3. Companies report 60–80% token cost reduction from context compression alone.

Batch APIs let you submit a collection of requests that are processed asynchronously within a 24-hour window. In exchange for accepting latency, you get 50% off the standard price. OpenAI, Anthropic, and Google all offer batch APIs. Implementation takes about 30 minutes — you submit a JSONL file of requests and poll for results.

Data processing (classify 10,000 documents overnight). Content generation (generate product descriptions in bulk). Evaluation (run test suites against model outputs). Embedding generation (batch-embed a document corpus). Any workload where results aren’t needed in real-time.

Batch discounts stack with prompt caching. If your batch requests share the same system prompt, you get both the 50% batch discount and the caching discount. On Anthropic, this means paying roughly 5% of the standard price for cached tokens in batch mode (50% batch × 10% cache rate).

Distillation trains a smaller, cheaper model to mimic the behavior of a larger, expensive model on your specific task. You generate training data by running your task through the expensive model, then fine-tune a small model on those outputs. The result: a model that performs 95–97% as well on your specific task at 5–30x lower cost.

Distillation makes sense when you have a well-defined, high-volume task that a premium model handles well. Customer support classification, code review, content moderation, and entity extraction are ideal candidates. It does not work well for open-ended creative tasks or tasks that require broad general knowledge.

DeepSeek demonstrated that aggressive distillation and efficiency optimization can produce models that compete with frontier models at 20–100x lower cost. DeepSeek V3.2 at $0.27/$0.42 per M tokens performs comparably to models costing 10–20x more on many benchmarks. This competitive pressure has forced all providers to improve efficiency.

A production AI gateway implementing routing + caching + batching achieves 47–80% total cost reduction. The first three optimizations (caching, batching, routing) can be implemented in a single day and typically deliver 40–60% savings. Adding compression and semantic caching pushes to 60–70%. Distillation, when applicable, can push individual task costs down by an additional 5–30x.

The KV-cache hit rate measures what percentage of your input tokens are served from cache vs recomputed from scratch. A high hit rate (80%+) means your caching strategy is working — most requests reuse previously computed token representations. A low hit rate (<30%) means you’re paying full price for every request.

Stabilize your prompt prefix. Don’t dynamically change system prompts, tool definitions, or few-shot examples between requests. Every change invalidates the cache. Order matters — put the most stable content (system prompt) first, variable content (user query) last. This maximizes the cacheable prefix length.

Chapter 7 applies everything we’ve learned to AI agents — the most expensive and hardest-to-optimize AI workload. The employee analogy: agents bill by the minute and sometimes spin in circles. Doom loops, monitoring, and the 4-layer cost governance framework.

The water conservation analogy: use less (compression), reuse (caching), switch sources (routing). Model routing saves 40–60%. Prompt caching saves 45–90% on prefixes in 10 minutes. Batch APIs give 50% off for async work. Distillation delivers 5–30x reduction on high-volume tasks. Combined: 60–70% typical savings. KV-cache hit rate is the key production metric.