The transformer's core innovation is self-attention: a mechanism that lets every token in a sequence look at every other token to determine how much each one matters for understanding the current token. Each token is projected into three vectors: Query (what am I looking for?), Key (what do I contain?), and Value (what information do I provide?). Attention scores are computed as the dot product of queries and keys, scaled and softmaxed into weights, then used to create a weighted sum of values. The result: each token's representation is enriched by information from the most relevant other tokens. Unlike RNNs that process sequentially, self-attention processes all tokens in parallel, enabling massive GPU parallelism. This is why transformers can be trained on billions of tokens — they're orders of magnitude faster than RNNs.

The full transformer architecture (Vaswani et al., 2017) stacks several components. Multi-head attention runs self-attention multiple times in parallel (typically 8–16 heads), each learning different relationship types — one head might learn syntactic dependencies, another semantic similarity. Feed-forward layers after each attention block add non-linear transformations, increasing the model's capacity. Layer normalization and residual connections stabilize training of deep networks. Positional encoding injects word order information, since self-attention is inherently order-agnostic — without it, "dog bites man" and "man bites dog" would have identical representations. The original transformer used sinusoidal positional encodings; modern models use learned positional embeddings or relative position encodings like RoPE.

The original transformer had both an encoder and decoder. Subsequent models discovered that using just one half works better for specific tasks. Encoder-only models (BERT, RoBERTa) use bidirectional attention — every token sees every other token. This is ideal for understanding tasks: classification, NER, question answering. Decoder-only models (GPT, LLaMA, Claude) use causal attention — each token can only see previous tokens. This is ideal for generation tasks and is the architecture behind all modern LLMs. Encoder-decoder models (T5, BART) use the encoder to understand the input and the decoder to generate the output. This is ideal for transformation tasks: translation, summarization, where you need to comprehend input before producing output.

BERT (Bidirectional Encoder Representations from Transformers, Google 2018) was the first model to demonstrate that a single pre-trained model could achieve state-of-the-art on 11 NLP benchmarks simultaneously. BERT is pre-trained with two objectives: Masked Language Modeling (MLM) — randomly mask 15% of tokens and predict them from context ("The [MASK] sat on the mat" → "cat"), and Next Sentence Prediction (NSP) — predict whether two sentences are consecutive. MLM forces bidirectional understanding: to predict a masked word, the model must use both left and right context. For downstream tasks, BERT adds a task-specific head (a linear layer) on top and fine-tunes the entire model. Classification uses the [CLS] token representation; NER uses per-token representations. This pre-train then fine-tune paradigm became the standard workflow for NLP.

GPT (Generative Pre-trained Transformer, OpenAI 2018) took the opposite approach from BERT: instead of bidirectional understanding, GPT uses causal (left-to-right) attention and pre-trains on next-token prediction. This makes GPT a natural text generator. GPT-1 (117M parameters) demonstrated that pre-training + fine-tuning works for generation. GPT-2 (1.5B) showed that scaling produces coherent multi-paragraph text. GPT-3 (175B) revealed emergent abilities: without any fine-tuning, it could perform tasks from just a few examples in the prompt (few-shot learning). This discovery — that scale alone produces new capabilities — launched the LLM revolution. The decoder-only architecture became the dominant paradigm: GPT-4, Claude, LLaMA, Gemini, and Mistral all use decoder-only transformers.

T5 (Text-to-Text Transfer Transformer, Google 2019) proposed a radical simplification: every NLP task is a text-to-text problem. Classification: "classify: This movie is great" → "positive." Translation: "translate English to German: Hello" → "Hallo." Summarization: "summarize: [long text]" → "[summary]." By framing all tasks identically, T5 uses the same model, loss function, and training procedure for everything. T5 uses an encoder-decoder architecture with span corruption pre-training: random spans of text are replaced with sentinel tokens, and the model learns to reconstruct them. The systematic study behind T5 tested dozens of design choices (model size, pre-training objective, data, fine-tuning strategy) and published the results, making it one of the most influential NLP papers for practical guidance.

The choice between BERT-style and GPT-style models depends on your task. BERT (encoder) excels at tasks that require understanding the full input: classification, NER, semantic similarity, extractive QA. Its bidirectional attention means every token can attend to every other token, producing richer representations for understanding. GPT (decoder) excels at generating text: dialogue, creative writing, code generation, and any task where you need to produce new text. Its causal attention naturally supports autoregressive generation. In practice, the distinction has blurred: modern LLMs (GPT-4, Claude) are so capable that they can perform understanding tasks through generation (classify by generating the label). But for production systems where efficiency matters, BERT-style models remain the better choice for classification and extraction tasks — they're 10–100x smaller and faster.

Transformers didn't just improve NLP — they unified it. Before transformers, each NLP task had its own specialized architecture. After transformers, one architecture handles everything. Three properties explain their dominance. Parallelism: unlike RNNs that process tokens sequentially, transformers process all tokens simultaneously, making them 10–100x faster to train on GPUs. Scalability: transformer performance improves predictably with model size, data size, and compute (scaling laws). This predictability enables billion-dollar training investments. Transfer learning: pre-training on massive unlabeled text produces representations that transfer to any downstream task with minimal fine-tuning. The transformer's impact extends far beyond NLP: Vision Transformers (ViT) for images, AlphaFold for protein structure, and multimodal models all use the same architecture.