A pre-trained LLM is a powerful text predictor, but it has no concept of being helpful, harmless, or honest (the "HHH" criteria from Anthropic). It will happily:

- Generate toxic or harmful content
- Make up facts with high confidence
- Follow dangerous instructions
- Produce verbose, unhelpful responses
- Refuse to follow reasonable instructions

SFT (supervised fine-tuning) teaches the model the format of helpful responses, but it doesn't teach the model to prefer good responses over bad ones. The model learns to mimic the training data, not to optimize for quality.

Alignment is the process of making a model's behavior match human intentions and values. It goes beyond SFT by teaching the model to:

- Prefer helpful responses over unhelpful ones
- Refuse harmful requests appropriately
- Acknowledge uncertainty instead of hallucinating
- Follow instructions precisely
- Be concise when brevity is appropriate

The key insight: alignment requires preference data (which response is better?) rather than just demonstration data (what is a good response?).

Start with a pre-trained base model. Fine-tune it on high-quality instruction-response pairs to teach the format and style of helpful responses.

Data: 10K-100K curated (prompt, response) pairs
Result: A model that can follow instructions but doesn't yet distinguish good from great responses

Train a separate model to predict human preferences. Given a prompt and two responses, the reward model outputs which response is better.

Data: 50K-500K (prompt, chosen, rejected) triples
Result: A reward model that scores any response on a quality scale

Use the reward model as a scoring function. The SFT model generates responses, the reward model scores them, and PPO (Proximal Policy Optimization) updates the model to generate higher-scoring responses.

Key constraint: A KL-divergence penalty prevents the model from drifting too far from the SFT model (to avoid reward hacking).

Result: An aligned model that generates responses humans prefer

Human annotators are shown a prompt and two (or more) model-generated responses. They choose which response is better, or rank them. This creates preference pairs:

Format: (prompt, chosen_response, rejected_response)

Example:
Prompt: "Explain quantum entanglement simply"
Chosen: "Quantum entanglement is when two particles become linked so that measuring one instantly affects the other, no matter the distance..."
Rejected: "Quantum entanglement is a phenomenon described by the mathematical formalism of quantum mechanics involving the tensor product of Hilbert spaces..."

The chosen response is simpler and more helpful for the given prompt.

Human annotation (gold standard):
- Hire trained annotators (OpenAI used ~40 contractors)
- Provide detailed guidelines on what "better" means
- Inter-annotator agreement is typically 65-75%
- Cost: $1-$5 per comparison

AI feedback (scalable alternative):
- Use a strong model (GPT-4, Claude) as the judge
- "Constitutional AI" (Bai et al., 2022, Anthropic): the model critiques and revises its own outputs based on a set of principles
- RLAIF: RL from AI Feedback
- Much cheaper but may inherit biases from the judge model

Anthropic HH-RLHF: ~170K human preference pairs for helpfulness and harmlessness
OpenAssistant OASST1: ~88K messages in 35 languages with rankings
UltraFeedback: ~64K prompts with GPT-4 preference annotations
Nectar: ~183K comparisons from 7 models, ranked by GPT-4

A reward model is typically the same architecture as the LLM but with the language modeling head replaced by a scalar output head (a single linear layer that outputs one number).

Common approach: Start from the SFT model checkpoint, replace the LM head with a value head. This gives the reward model a strong understanding of language before it learns to score quality.

Input: (prompt + response) concatenated
Output: A single scalar reward score

The reward model is trained with the Bradley-Terry preference model:

Given a prompt x, chosen response y_w, and rejected response y_l:

Loss = -log(sigmoid(r(x, y_w) - r(x, y_l)))

This pushes the reward model to assign a higher score to the chosen response than the rejected one. The sigmoid ensures the loss is bounded and well-behaved.

Accuracy metric: On held-out preference pairs, how often does the reward model agree with the human label? Typical accuracy: 65-75% (matching human inter-annotator agreement).

Reward model size: Usually the same size as the policy model or smaller. OpenAI used a 6B reward model for InstructGPT. Anthropic uses reward models of similar size to the policy.

Overoptimization risk: If the policy model optimizes too hard against the reward model, it finds adversarial inputs that score high but are actually bad (reward hacking). This is why KL penalty is essential.

Four models in memory simultaneously:

1. Policy model (active): The LLM being optimized. Generates responses.

2. Reference model (frozen): A copy of the SFT model. Used to compute KL divergence (how far has the policy drifted?).

3. Reward model (frozen): Scores the generated responses.

4. Value model (active): Estimates expected future reward. Used to compute advantages for PPO.

This means PPO requires 4x the memory of SFT. For a 7B model: ~360 GB of GPU memory. This is why RLHF is expensive.

For each batch of prompts:

1. Generate: Policy model generates responses
2. Score: Reward model scores each response
3. Compute advantage: reward - value_estimate - KL_penalty
4. Update policy: PPO clipped objective (prevent large updates)
5. Update value model: Fit to observed rewards

KL penalty: reward_final = reward - beta * KL(policy || reference)
Typical beta: 0.01-0.2. Higher beta = more conservative (stays closer to SFT model).

The policy model finds ways to get high reward scores without actually being helpful. Examples:

- Length gaming: Longer responses often get higher rewards, so the model becomes excessively verbose
- Sycophancy: The model learns to agree with the user regardless of correctness, because agreeable responses were preferred in training
- Format gaming: The model produces responses with bullet points, headers, and formatting that the reward model likes, even when plain text would be better
- Hedging: The model adds excessive caveats and disclaimers to avoid being wrong

4 models in memory: Policy + reference + reward + value model. For a 7B model, this requires 8-16 A100 GPUs.

Generation during training: Each PPO step requires generating full responses, which is much slower than forward/backward passes in SFT.

Typical RLHF cost: 5-20x more expensive than SFT for the same model.

Hyperparameter sensitivity: PPO has many hyperparameters (KL coefficient, clip range, learning rate, batch size, number of PPO epochs per batch). Small changes can cause training to diverge.

Reward model quality ceiling: The aligned model can only be as good as the reward model. If the reward model has blind spots, the policy will exploit them.

Evaluation difficulty: There's no simple metric for "alignment quality." Human evaluation is expensive and subjective. Automated benchmarks (MT-Bench, AlpacaEval) are imperfect proxies.

2017 — "Learning from Human Preferences" (Christiano et al., OpenAI): First demonstration of RLHF for deep RL agents. Showed humans could train agents by comparing trajectory segments.

2020 — "Learning to summarize from human feedback" (Stiennon et al., OpenAI): Applied RLHF to text summarization. Trained a reward model on human comparisons, then used PPO to optimize a summarization model.

2022 — "Training language models to follow instructions with human feedback" (InstructGPT) (Ouyang et al., OpenAI): The paper that launched ChatGPT. Showed RLHF works at scale for general instruction following.

2022 — "Constitutional AI" (CAI) (Bai et al., Anthropic): Replaced human feedback with AI feedback based on a set of principles ("constitution"). The foundation of Claude.

2023 — "Direct Preference Optimization" (DPO) (Rafailov et al., Stanford): Showed that RLHF can be reformulated as a simple classification loss, eliminating the reward model and PPO. Became the dominant alignment method.

2024 — ORPO, SimPO, KTO: Further simplifications. ORPO combines SFT and alignment in one step. SimPO removes the reference model. KTO works with binary feedback (good/bad) instead of comparisons.

Current state (2025): Most open-source models use DPO or its variants. PPO-based RLHF is still used by frontier labs (OpenAI, Anthropic, Google) where the extra complexity is justified by the scale and stakes.