Recall from Chapter 7: the seq2seq encoder compresses an entire input sentence into a single fixed-size vector. For short sentences, this works. But for a 50-word sentence, cramming all meaning into a 512-dimensional vector is like summarizing a novel in a tweet. Information about early words gets overwritten by later ones. Translation quality degrades sharply for sentences longer than ~20 words. The fundamental problem: the decoder has equal access to all parts of the input (via the context vector) but no way to focus on specific parts when generating each output word.

Bahdanau, Cho, and Bengio (ICLR 2015) proposed that instead of compressing the entire input into one vector, the decoder should attend to different parts of the input at each step. At each decoding step, the model computes attention weights — a probability distribution over all encoder hidden states — indicating which input words are most relevant for the current output word. The context vector is then a weighted sum of all encoder states, dynamically changing at each step. When generating “L’accord,” the model attends heavily to “The agreement.”

Vaswani et al. (2017) generalized attention using a database analogy. A query (Q) is what you’re looking for. Keys (K) are labels on stored items. Values (V) are the actual stored content. Attention computes the similarity between the query and each key, then returns a weighted sum of values. The similarity is computed as a scaled dot product: score = Q·K¹/√d_k, where d_k is the key dimension. The scaling prevents dot products from growing too large in high dimensions, which would push softmax into regions with tiny gradients.

Bahdanau attention is cross-attention: the decoder attends to the encoder. Self-attention is when a sequence attends to itself. Each position computes Q, K, V from its own representation, then attends to all other positions in the same sequence. This lets every word in a sentence directly interact with every other word, regardless of distance. In “The cat sat on the mat because it was tired,” self-attention lets “it” directly attend to “cat” to resolve the pronoun — no need to pass information through intermediate words.

A single attention head can only focus on one type of relationship at a time. Multi-head attention runs h parallel attention operations (heads), each with its own learned Q, K, V projections. Different heads learn to attend to different things: one head might capture syntactic relationships (subject-verb), another semantic similarity, another positional patterns. The outputs of all heads are concatenated and projected back to the model dimension. GPT-3 uses 96 heads; GPT-4 is estimated to use 128+.

Research on BERT and GPT attention patterns reveals that different heads specialize in different linguistic relationships: positional heads attend to adjacent tokens, syntactic heads connect subjects to verbs, coreference heads link pronouns to their antecedents, and delimiter heads attend to special tokens like [SEP] or [CLS]. Some heads appear to be “no-op” heads that can be pruned without affecting performance. This emergent specialization happens without any explicit supervision.

Self-attention computes a score between every pair of positions, creating an n×n attention matrix. For a sequence of length n, this is O(n²) in both time and memory. A 1,000-token sequence needs 1 million attention scores; 100,000 tokens needs 10 billion. This quadratic cost is why early transformers were limited to 512–2048 tokens. Solutions include sparse attention (attend to only nearby + selected distant positions), linear attention (approximate softmax), FlashAttention (GPU memory-efficient exact attention), and sliding window attention (used in Mistral).

Attention started as an add-on to RNNs (Bahdanau, 2015). Then Vaswani et al. (2017) asked the radical question: what if we use only attention, with no recurrence at all? The answer was the Transformer — an architecture built entirely from self-attention and feedforward layers. It was faster to train (fully parallelizable), handled long-range dependencies better (O(1) path length), and achieved state-of-the-art results on translation. Every major AI model since — BERT, GPT, LLaMA, Gemini, Claude — is a Transformer.