The context window is the maximum amount of text (measured in tokens) that an LLM can process in a single request. It includes everything: system instructions, user input, conversation history, retrieved documents, tool definitions, and the model’s own output. Think of it as the model’s working memory — anything outside the window simply doesn’t exist to the model.

1. System prompt — Instructions that define the model’s role, behavior, and constraints. Typically 200–2,000 tokens.

2. User prompt — The actual question or task from the user. Usually the smallest component in production systems.

3. Conversation history — Prior turns in the dialogue. Grows linearly with conversation length and is the primary source of context bloat.

4. Retrieved documents (RAG) — External knowledge fetched via search or retrieval. Can be the largest single component, often 10,000–50,000+ tokens.

5. Tool definitions — JSON schemas describing available tools/APIs. A single complex schema can consume 500+ tokens. 90 tools means 50K+ tokens before any user interaction.

6. Few-shot examples — Input/output pairs that demonstrate desired behavior. High-quality examples improve consistency but consume significant space.

7. Memory stores — Persisted facts from previous sessions (user preferences, past decisions). Enables continuity across conversations.

8. Metadata — Timestamps, user IDs, session state, environment variables. Small individually but adds up across complex agent systems.

Research has consistently shown that LLMs exhibit a U-shaped performance curve across the context window. Models attend strongly to content at the beginning (primacy effect) and end (recency effect) of the context, but struggle with information placed in the middle. This is the “lost in the middle” phenomenon, first documented in 2023 and still present in 2026 models despite larger windows.

The standard evaluation: insert a specific fact at various positions in a long context and test whether the model can retrieve it. Results show systematic blind spots in the middle third of the context window. Even models advertising 200K+ token windows show degraded retrieval accuracy for middle-positioned information.

Context engineering addresses this by controlling information placement: put critical instructions at the beginning (system prompt), put the most relevant retrieved content near the end (close to the query), and use compression to eliminate middle-section noise. Systematic context management can prevent 30% of information loss from this effect.

Most models experience meaningful performance degradation 30–40% before their advertised context limit. A model with a 128K token window may start producing noticeably worse output around 80K–90K tokens. This isn’t a bug — it’s a fundamental property of how attention mechanisms scale with sequence length.

The computational cost of attention scales quadratically with context length. Doubling the context doesn’t just double the cost — it quadruples it. This means cost escalation is geometric, and strategic compression and caching become essential at scale.

Including irrelevant information actively degrades model attention on essential tokens. It’s not neutral — it’s harmful. Every irrelevant document, every stale conversation turn, every unused tool definition is competing for the model’s finite attention budget. The core principle of context engineering is “Select, Don’t Dump” — include only what’s necessary for the next step.

The KV-cache (Key-Value cache) stores pre-computed attention states from the transformer’s self-attention mechanism. When generating tokens, the model doesn’t need to recompute attention for the entire context from scratch — it reuses cached key-value pairs from previous tokens. This is what makes autoregressive generation fast enough to be practical.

Intra-request caching: Within a single generation, previously computed KV pairs are reused for each new token. This is automatic and universal.

Prefix/prompt caching: Across requests, server-side features (available in Anthropic, Groq, and vLLM) cache the KV states for shared prefixes. If your system prompt is the same across requests, it’s computed once and reused.

Industry leaders describe KV-cache hit rate as “the single most important metric” for production AI agents. High cache hit rates can reduce inference latency and costs by 10×. The key optimization: separate stable prefix content (system instructions, tool definitions) from dynamic content (user queries, retrieved docs) to maximize cache reuse.

A fintech startup reduced document analysis costs from $30,600 to $4,100 monthly (87% reduction) through three context engineering techniques: extracting only relevant document sections via RAG instead of including full documents, compressing conversation history with sliding window summarization, and caching system prompts using provider-level prompt caching.

Newer model providers offer prompt caching that provides up to 90% cost savings on repeated content. System prompts and few-shot examples become nearly free on repeat calls. This makes the architectural decision of what goes in the stable prefix vs. dynamic content a direct cost optimization lever.

The first portion of the context window should contain stable, rarely-changing content: system instructions, tool definitions, and few-shot examples. This maximizes KV-cache hit rates across requests and ensures the model always has its core instructions in the high-attention “primacy” zone.

Conversation history, retrieved documents, and memory stores occupy the middle. This is where compression and routing do their work — keeping only the most relevant turns, selecting only the most pertinent documents, and summarizing older context to preserve space.

The most recent user query and the most relevant retrieved content should be placed near the end of the context, in the high-attention “recency” zone. This exploits the U-shaped attention curve to ensure the model focuses on what matters most for the current task.

AI agents using the ReAct pattern (Reason + Act) accumulate context with every tool call. Each cycle adds the tool result (hundreds or thousands of tokens), the model’s reasoning, and the action taken. Without intervention, a 10-step agent task can consume the entire context window — pushing out the system instructions and early task context the model needs to reason well.

In multi-agent systems, each agent has its own context window. Information must be selectively routed between agents — a billing agent doesn’t need the onboarding knowledge base, and a code review agent doesn’t need the deployment logs. Without routing, every agent carries every piece of context, multiplying waste.

The remaining chapters of this course address each dimension of this challenge: progressive disclosure (Ch 3) controls what loads and when, compression (Ch 4) shrinks accumulated history, routing (Ch 5) directs queries to the right source, retrieval (Ch 6) fetches external knowledge on demand, tool management (Ch 7) controls the capability surface, and token budgeting (Ch 8) ties it all together economically.