LLMs don't see text. They see sequences of integers (token IDs). The tokenizer converts text into these IDs and back. Each model has its own tokenizer with a fixed vocabulary. Llama 3 uses a vocabulary of 128,256 tokens. GPT-4o uses approximately 200,000 tokens.

Tokenizers use Byte-Pair Encoding (BPE) or SentencePiece. Common words become single tokens; rare words are split into subword pieces. "unhappiness" might become ["un", "happiness"] or ["un", "happ", "iness"].

Models use special tokens to mark structure:

Llama 3: <|begin_of_text|>, <|start_header_id|>, <|end_header_id|>, <|eot_id|>
ChatML: <|im_start|>, <|im_end|>
Mistral: [INST], [/INST]

These tokens tell the model where system prompts, user messages, and assistant responses begin and end. Getting them wrong during fine-tuning causes garbled output.

1. Chat templates: Your training data must use the exact same special token format the model expects. Mismatched templates are the #1 cause of bad fine-tuning results.

2. Vocabulary size: The tokenizer's vocabulary is fixed. You can't add new tokens easily (though it's possible with embedding resizing). If your domain has specialized terms, the tokenizer will split them into subwords.

3. Token count: Tokenization determines training cost. A 1,000-word document might be 1,200-1,500 tokens. Training cost is proportional to total tokens processed.

The embedding layer is a lookup table: a matrix of shape (vocab_size, hidden_dim). For Llama 3 8B: (128256, 4096) = 525 million parameters. Each token ID maps to a 4096-dimensional vector.

These vectors encode semantic meaning. Similar words have similar vectors. "king" and "queen" are close in embedding space; "king" and "bicycle" are far apart.

Transformers have no built-in sense of word order. Positional encoding adds position information to the embeddings.

RoPE (Rotary Position Embedding) is used by Llama 3, Mistral, and most modern models. It encodes position by rotating the embedding vectors. This allows the model to generalize to longer sequences than it was trained on (with some quality loss).

The embedding layer is typically not fine-tuned with LoRA (it's already well-trained during pre-training). In full fine-tuning, embeddings change slightly but they represent a small fraction of total parameters (~6% for Llama 3 8B).

If you add new special tokens (e.g., for a custom chat template), you need to resize the embedding matrix and initialize the new token embeddings. The new embeddings start random and need training data to learn meaningful representations.

For each token, attention asks: "Which other tokens should I pay attention to?" It computes three vectors from each token's representation:

Query (Q): "What am I looking for?"
Key (K): "What do I contain?"
Value (V): "What information do I provide?"

Attention score = softmax(Q · K^T / √d). High scores mean strong relevance. The output is a weighted sum of V vectors, where weights are the attention scores.

Instead of one attention computation, the model runs multiple attention heads in parallel. Each head learns different relationship patterns: one head might track subject-verb agreement, another tracks coreference, another tracks semantic similarity.

Llama 3 8B: 32 attention heads, each with dimension 128 (32 × 128 = 4096).
Llama 3 70B: 64 attention heads, each with dimension 128 (64 × 128 = 8192).

Standard multi-head attention has separate Q, K, V projections per head. GQA (Ainslie et al., 2023) shares K and V across groups of query heads. This reduces memory during inference (smaller KV cache) with minimal quality loss.

Llama 3 8B: 32 query heads, 8 KV heads (4:1 ratio).
Llama 3 70B: 64 query heads, 8 KV heads (8:1 ratio).
Mistral 7B: 32 query heads, 8 KV heads.

For autoregressive generation (predicting the next token), each token can only attend to previous tokens, not future ones. This is enforced by a causal mask: a triangular matrix that sets future attention scores to negative infinity before softmax.

After attention mixes information between tokens, the FFN processes each token independently. It's a two-layer neural network with a non-linear activation in between:

FFN(x) = W2 · activation(W1 · x)

The FFN expands the hidden dimension (4096 → 14336 for Llama 3 8B), applies a non-linearity, then projects back down (14336 → 4096). This expansion gives the model more capacity to learn complex transformations.

Modern LLMs use SwiGLU (Shazeer, 2020) instead of ReLU. SwiGLU uses a gating mechanism: it multiplies two linear projections element-wise, with one passed through a Swish activation. This requires three weight matrices instead of two (gate_proj, up_proj, down_proj) but produces better results.

Llama 3 8B FFN: gate_proj (4096 × 14336), up_proj (4096 × 14336), down_proj (14336 × 4096) = 176M parameters per layer.

The FFN layers contain the majority of parameters in a transformer. For Llama 3 8B:

Attention per layer: Q (4096×4096) + K (4096×1024) + V (4096×1024) + O (4096×4096) = 41.9M params

FFN per layer: gate (4096×14336) + up (4096×14336) + down (14336×4096) = 176.2M params

FFN is 4.2x larger than attention per layer. Across 32 layers: attention = 1.34B, FFN = 5.64B. The FFN is where most of the model's "knowledge" is stored.

Each transformer layer contains:

1. RMSNorm (pre-normalization)
2. Multi-Head Attention (with GQA)
3. Residual connection (add input back)
4. RMSNorm (pre-normalization)
5. Feed-Forward Network (SwiGLU)
6. Residual connection (add input back)

This block repeats N times. The residual connections are critical: they allow gradients to flow through the network without vanishing, enabling very deep models.

Llama 3 8B: 32 layers, hidden_dim=4096
Llama 3 70B: 80 layers, hidden_dim=8192
Llama 3 405B: 126 layers, hidden_dim=16384
Mistral 7B: 32 layers, hidden_dim=4096
Phi-3 Mini (3.8B): 32 layers, hidden_dim=3072
Qwen 2.5 72B: 80 layers, hidden_dim=8192

More layers = more capacity but more memory and slower inference.

Research shows different layers specialize:

Early layers (1-10): Low-level features. Syntax, grammar, local word relationships. These change least during fine-tuning.

Middle layers (11-22): Semantic understanding. Meaning, context, reasoning patterns. These change moderately.

Late layers (23-32): Task-specific features. Output formatting, style, domain-specific patterns. These change most during fine-tuning.

This is why LoRA often works well: you only need to modify the upper layers' behavior, and small adapters are sufficient for that.

The LM head takes the final hidden state (a 4096-dimensional vector for Llama 3 8B) and projects it to vocabulary size (128,256 dimensions). Each dimension represents the logit (unnormalized score) for one token in the vocabulary.

Apply softmax to convert logits to probabilities. The token with the highest probability is the model's prediction for the next token. During generation, this process repeats autoregressively.

Temperature: Divides logits before softmax. T=0.0 is greedy (always pick the highest). T=1.0 is the trained distribution. T>1.0 makes output more random.

Top-p (nucleus sampling): Only sample from the smallest set of tokens whose cumulative probability exceeds p (e.g., 0.9).

Top-k: Only consider the k most likely tokens.

These are inference-time parameters. They don't affect fine-tuning, but understanding them helps you evaluate your fine-tuned model.

The LM head is a single linear layer: (hidden_dim, vocab_size). For Llama 3 8B: (4096, 128256) = 525M parameters.

With weight tying, this matrix is the same as the embedding matrix (transposed). So the model's first and last layers share parameters. This is standard in Llama 3, Mistral, and most modern models.

The LM head is typically not targeted by LoRA. It's already well-trained and shares weights with the embedding layer.

safetensors: The modern standard (HuggingFace). Safe, fast, memory-mapped. Used by Llama 3, Mistral, and most open models. Files are named model-00001-of-00004.safetensors for sharded models.

PyTorch (.bin): Legacy format using Python pickle. Can execute arbitrary code on load (security risk). Being phased out in favor of safetensors.

GGUF: Format used by llama.cpp and Ollama. Optimized for CPU inference and quantized models. Single file containing model + metadata + tokenizer.

fp32 (32-bit float): Full precision. 4 bytes per parameter. 7B model = 28 GB. Used for optimizer states.

fp16 (16-bit float): Half precision. 2 bytes per parameter. 7B = 14 GB. Standard for training and inference.

bf16 (bfloat16): Same size as fp16 but with fp32's exponent range. Better for training (less overflow/underflow). Preferred on A100/H100 GPUs.

int8 (8-bit integer): 1 byte per parameter. 7B = 7 GB. Quantized inference with minimal quality loss.

int4 (4-bit integer): 0.5 bytes per parameter. 7B = 3.5 GB. Used by QLoRA (NF4 format). Some quality loss but enables fine-tuning large models on consumer GPUs.

GPTQ: Post-training quantization. Calibrates on a small dataset. Good quality at 4-bit. Used for inference deployment.

AWQ (Activation-Aware Weight Quantization): Preserves salient weights based on activation patterns. Often slightly better than GPTQ.

NF4 (Normal Float 4-bit): Used by QLoRA. Quantization-aware format optimized for normally-distributed weights. Enables 4-bit base model loading during fine-tuning.

GGUF quantization: Multiple levels (Q2_K through Q8_0). Q4_K_M is the most popular balance of quality and size for local inference.