In June 2017, eight Google researchers published a paper with a provocative title: “Attention Is All You Need.” They proposed the Transformer — a sequence-to-sequence model built entirely from attention mechanisms, with no recurrence and no convolutions. It achieved 28.4 BLEU on English-to-German translation (a new state of the art) while training in just 3.5 days on 8 GPUs — far faster than RNN-based models. The Transformer’s parallelism and scalability made it the foundation for every major AI breakthrough since: BERT, GPT, LLaMA, Gemini, Claude, and beyond.

Self-attention is permutation-invariant — it treats “cat sat mat” and “mat cat sat” identically because it only computes pairwise similarities, not positions. But word order matters! Positional encodings add position information to the input embeddings. The original Transformer used sinusoidal encodings: each position gets a unique pattern of sine and cosine waves at different frequencies. Modern models (GPT, LLaMA) use learned positional embeddings or Rotary Position Embeddings (RoPE) which encode relative positions directly in the attention computation.

Each Transformer layer (block) has two sub-layers: multi-head self-attention and a position-wise feedforward network (FFN). Each sub-layer has a residual connection (like ResNet’s skip connections) and layer normalization. The FFN is a simple 2-layer MLP applied independently to each position: Linear(d_model → d_ff) → GELU → Linear(d_ff → d_model). The FFN is where the model stores “knowledge” — factual associations learned during training. The attention layers handle relationships between positions.

In language generation, the model predicts one token at a time, left to right. When predicting token 5, it must not see tokens 6, 7, 8... This is enforced with a causal mask (also called an attention mask): a triangular matrix that sets attention scores to -∞ for future positions, making their softmax weights zero. This ensures each position can only attend to itself and earlier positions. BERT uses no mask (bidirectional); GPT uses a causal mask (autoregressive). This is the fundamental difference between encoder-only and decoder-only architectures.

Encoder-only (BERT): bidirectional self-attention, no causal mask. Excels at understanding tasks (classification, NER, similarity). Decoder-only (GPT): causal self-attention, generates text left-to-right. Dominates language generation and has become the default for LLMs. Encoder-decoder (T5, original Transformer): encoder processes input bidirectionally, decoder generates output autoregressively with cross-attention to the encoder. Used for translation, summarization.

Kaplan et al. (OpenAI, 2020) discovered that Transformer performance follows power laws: loss decreases predictably as you increase model size, dataset size, and compute. Double the parameters → predictable improvement. This means you can predict how well a model will perform before training it, enabling efficient resource allocation. The Chinchilla scaling laws (Hoffmann et al., 2022) refined this: for a given compute budget, there’s an optimal ratio of model size to training tokens (~20 tokens per parameter).

The core Transformer design has been refined significantly: Grouped-Query Attention (GQA) shares key/value heads across query heads, reducing memory during inference. SwiGLU activation (Shazeer, 2020) replaces ReLU/GELU in the FFN for better performance. RMSNorm replaces LayerNorm for efficiency. Pre-norm (normalize before attention) replaces post-norm for more stable training. KV caching stores computed key/value pairs to avoid recomputation during autoregressive generation. These are all used in LLaMA, Mistral, and most modern LLMs.

This course traced a path from the 1943 McCulloch-Pitts neuron to the 2017 Transformer. Every concept builds on the last: perceptrons → backpropagation → CNNs (spatial hierarchy) → RNNs (sequence memory) → LSTMs (gating) → attention (direct connections) → Transformers (attention is all you need). The Transformer combined the best ideas from 74 years of neural network research into a single, scalable architecture. GPT, BERT, LLaMA, Claude, Gemini — they are all Transformers, trained on the same principles covered in this course.