When you type in your editor, the AI coding tool captures raw text — your current file, cursor position, and surrounding context. To the model, Python, TypeScript, and Rust are all just sequences of characters. There’s no special “code mode” — the model processes code the same way it processes English, through token prediction.

The tool assembles a prompt from your code. For inline completions, this typically includes the prefix (code before your cursor) and suffix (code after your cursor). For chat, it includes your question plus relevant file context. This assembled text is the model’s entire view of your world.

Code has properties that make it distinct from prose: strict syntax (a missing bracket breaks everything), semantic indentation (Python, YAML), cross-file references (imports, types), and execution semantics (the code must actually run). The model doesn’t “understand” any of this — it learns statistical patterns that happen to capture these structures.

Code LLMs use BPE tokenizers (originally a 1994 compression algorithm). BPE starts with individual bytes (256 base tokens) and iteratively merges the most frequent adjacent pairs. Common words like function or return become single tokens. Rare identifiers like calculateMonthlyRevenue get split into subwords: calculate + Monthly + Revenue.

English text averages 1–1.5 tokens per word. Code averages 1.5–4 tokens per word because of special characters, camelCase splitting, indentation, and punctuation like {, =>, ::. This means code fills up context windows faster than English — a 128K-token window holds less code than you’d expect.

Each token ID is mapped to a high-dimensional vector (typically 4,096–12,288 dimensions in modern code models). This embedding captures the token’s learned “meaning” — function and def end up near each other in vector space because they appear in similar contexts across training data.

Order matters in code — a = b is different from b = a. Positional encodings (like RoPE, used in most modern models) are added to each embedding so the model knows where each token sits in the sequence, not just what it is.

At this stage, your code is a matrix of numbers — each row is one token’s embedding vector. A 500-token code snippet becomes a 500 × 4,096 matrix. The model has no concept of “Python” or “function” — just patterns of numbers that it will process through transformer layers.

The transformer’s self-attention mechanism lets every token “look at” every other token in the context. When predicting the next token after items.reduce((sum, item) => sum +, the model attends heavily to item, reduce, and the variable names — learning that this pattern typically ends with a property access like item.price.

Modern code models have 32–80+ transformer layers. Early layers capture syntax (bracket matching, indentation). Middle layers capture semantics (variable types, function signatures). Deep layers capture high-level patterns (design patterns, algorithm structure). Each layer refines the representation.

The context window is the maximum number of tokens the model can process at once. GPT-4o supports 128K tokens, Claude supports 200K tokens. This determines how much of your codebase the model can “see” in a single request. Anything outside the window simply doesn’t exist to the model.

After passing through all transformer layers, the model outputs a probability score for every token in its vocabulary (typically 32K–128K tokens). For code completion, the top candidates might be: item.price (72%), item.cost (15%), item.amount (8%), with thousands of other tokens sharing the remaining 5%.

Temperature controls how “creative” the model is. At temperature 0 (greedy), it always picks the highest-probability token — deterministic and safe. At temperature 0.7–1.0, lower-probability tokens get a chance, producing more varied output. For code completion, tools typically use low temperature (0–0.2) because correctness matters more than creativity.

Top-p sampling only considers tokens whose cumulative probability reaches a threshold (e.g., p=0.95). This dynamically adjusts the candidate pool — when the model is confident, only 2–3 tokens qualify. When uncertain, dozens might. It’s more adaptive than a fixed top-k cutoff.

The model generates one token per forward pass. To produce a 10-line function, it runs hundreds of sequential predictions, each time feeding its previous output back as input. This is why generation has noticeable latency — and why speculative decoding (predicting multiple tokens at once) is a major optimization.

Standard LLMs generate text left-to-right — they can only continue from the end. But coding often requires inserting code in the middle: adding a function body between its signature and the next function, filling in a parameter list, or completing a block inside existing logic.

Fill-in-the-Middle (FIM) uses special tokens to split code into prefix, middle, and suffix. The model sees the code before your cursor (prefix) and after your cursor (suffix), then generates the missing middle. This is the core technique behind inline code completion.

Every request has a token budget. The tool must decide what to include: current file, open tabs, imported files, type definitions, recent edits, conversation history. It’s a packing problem — fitting the most useful context into a fixed window. What gets left out is invisible to the model.

Modern tools pull context from multiple sources:

• Current file — code around the cursor (highest priority)
• Open tabs — files you’re actively working with
• Codebase index — semantic search across the full project
• Type definitions — interfaces, schemas, API contracts
• Recent edits — what you just changed (intent signal)

Research shows LLMs attend most strongly to tokens at the beginning and end of the context window, with weaker attention to the middle. This means the order of context matters — the most important code should be placed at the start or end of the prompt, not buried in the middle.

1. You type a keystroke
2. The tool extracts prefix + suffix around your cursor
3. It assembles context from open files, codebase index, and type definitions
4. The prompt is tokenized via BPE into integer sequences
5. Tokens are embedded into vectors with positional encoding
6. The transformer processes all tokens through 32–80+ attention layers
7. Output probabilities are sampled at low temperature
8. Generated tokens are decoded back to text and displayed as ghost text

This entire pipeline must complete in under 200ms to feel responsive. Techniques like speculative decoding (predicting multiple tokens using a smaller draft model), KV-cache reuse (not recomputing unchanged prefix tokens), and quantization (running in 4-bit or 8-bit precision) make this possible.

The model never executes your code, never checks types, never runs tests, and never verifies imports exist. It predicts the most statistically likely next tokens based on patterns in its training data. When it suggests import pandas as pd, it’s not checking if pandas is installed — it’s predicting what usually follows the patterns it’s seen.