A language model assigns probabilities to sequences of words. Given a context, it predicts what comes next. "The cat sat on the ___" — a good language model assigns high probability to "mat" and low probability to "democracy." Formally, a language model computes P(w₁, w₂, ..., w_n) — the probability of a sequence. By the chain rule, this decomposes into: P(w₁) × P(w₂|w₁) × P(w₃|w₁,w₂) × ... Language models are the foundation of modern NLP. Spell checkers, autocomplete, machine translation, speech recognition, and every LLM from GPT to Claude are language models at their core. The entire history of NLP can be told through the evolution of language models: from counting word sequences to neural networks that generate human-quality text.

N-gram models estimate the probability of a word based on the previous N−1 words. A bigram model uses one word of context: P("mat" | "the"). A trigram uses two: P("mat" | "on the"). Training is just counting: P("mat" | "on the") = count("on the mat") / count("on the"). N-gram models are fast, simple, and surprisingly effective for tasks like spell checking and speech recognition. But they have fundamental limitations. Sparsity: most n-grams never appear in training data, so their probability is zero. Smoothing techniques (Laplace, Kneser-Ney) redistribute probability mass to unseen n-grams. Limited context: even a 5-gram model only sees 4 previous words, missing long-range dependencies like "The doctor who treated the patient in the emergency room last Tuesday ___."

Perplexity is the standard metric for language models. Intuitively, it measures how "surprised" the model is by the test data. A perplexity of 100 means the model is as uncertain as if it were choosing uniformly among 100 words at each step. Lower perplexity = better model. Formally, perplexity is 2 raised to the power of the average negative log-probability: PPL = 2^{−(1/N) ∑ log₂ P(w_i|context)}. A perfect model that always predicts the correct next word has perplexity 1. A model that assigns equal probability to a 50,000-word vocabulary has perplexity 50,000. Modern LLMs achieve perplexities of 10–30 on standard benchmarks, meaning they effectively narrow down the next word to 10–30 candidates. Perplexity is useful for comparing models on the same test set but doesn't directly measure generation quality.

Neural language models replaced counting with learned representations. Bengio's 2003 neural language model used a feed-forward network over word embeddings — the first to show that neural networks could outperform n-grams. Recurrent Neural Networks (RNNs) process text one word at a time, maintaining a hidden state that summarizes everything seen so far. Unlike n-grams, RNNs have no fixed context window — in theory, they can use the entire preceding text. In practice, vanilla RNNs suffer from the vanishing gradient problem: information from early words fades as the sequence grows. LSTMs (Long Short-Term Memory) solved this with gating mechanisms that control what to remember and what to forget. LSTM language models achieved perplexities of 50–70, cutting n-gram perplexity nearly in half.

A language model becomes a text generator through autoregressive generation: predict the next word, add it to the context, predict the next word again, repeat. Given "The cat", the model predicts "sat" (highest probability), yielding "The cat sat". Then it predicts "on" from "The cat sat", and so on. This simple loop is how every modern text generator works, from GPT to Claude. The quality of generated text depends on two things: the quality of the language model (how well it predicts) and the decoding strategy (how it chooses from the predicted distribution). Greedy decoding always picks the most probable word, but this produces repetitive, boring text. Better strategies introduce controlled randomness to produce diverse, natural-sounding output.

The language model outputs a probability distribution over the vocabulary at each step. The decoding strategy determines how to select from this distribution. Greedy decoding always picks the highest-probability token — fast but produces repetitive, generic text. Beam search maintains the top-k candidate sequences at each step, finding globally better sequences than greedy — good for translation and summarization but still tends toward safe, boring output. Temperature sampling scales the logits before softmax: temperature < 1 makes the distribution sharper (more deterministic), temperature > 1 makes it flatter (more random). Top-k sampling restricts sampling to the k most probable tokens. Top-p (nucleus) sampling dynamically selects the smallest set of tokens whose cumulative probability exceeds p, adapting the candidate pool to model confidence.

Text generation faces several persistent challenges. Repetition: models tend to repeat phrases or sentences, especially with greedy/beam search. Repetition penalties and n-gram blocking help but don't eliminate the problem. Hallucination: models generate plausible-sounding but factually incorrect text because they optimize for probability, not truth. Coherence degradation: as generated text grows longer, it tends to drift off-topic or contradict earlier statements because the model has no explicit memory of its own output beyond the context window. Exposure bias: during training, the model sees gold-standard context; during generation, it sees its own (potentially wrong) predictions, causing errors to compound. Evaluation difficulty: there's no single metric that captures generation quality — fluency, coherence, factuality, and relevance are all separate dimensions.

The path from n-gram language models to GPT-4 and Claude is a story of scale. Each generation of language model was bigger, trained on more data, and surprisingly more capable. GPT-1 (2018, 117M parameters) showed that pre-trained language models could be fine-tuned for downstream tasks. GPT-2 (2019, 1.5B) demonstrated that larger models could generate remarkably coherent text. GPT-3 (2020, 175B) revealed emergent abilities: few-shot learning, basic reasoning, and code generation appeared without explicit training. The scaling laws (Kaplan et al., 2020) showed that language model performance improves predictably with model size, data size, and compute. This insight — that next-word prediction at sufficient scale produces general intelligence — is the foundation of the modern AI revolution. The next chapter covers the transformer architecture that made this scaling possible.