When you see “Llama-3.1-8B,” the 8B means 8 billion parameters — 8 billion individual numbers (weights) that the model learned during training. More parameters generally means more capacity to store knowledge and handle complex reasoning. The common sizes you’ll encounter: 1B–3B (edge/mobile), 7B–8B (single GPU sweet spot), 13B–14B (strong single-GPU), 32B–70B (multi-GPU or quantized), 405B+ (cluster-scale).

Parameter count directly determines how much memory (VRAM/RAM) you need. Each parameter at full precision (FP16) takes 2 bytes. So an 8B model needs ~16GB of VRAM just for the weights, plus overhead for KV-cache and activations. A 70B model needs ~140GB at FP16 — that’s multiple high-end GPUs. This is why quantization exists: an 8B model at 4-bit precision needs only ~4GB.

In a dense model, every parameter is used for every input token. Llama 3.1 8B is dense — all 8 billion parameters fire on every forward pass. This is simple but expensive at scale: double the parameters, double the compute.

In a Mixture-of-Experts model, the parameters are split into “expert” sub-networks, and a router selects only a few experts per token. Mixtral 8x7B has 46.7B total parameters but only ~12.9B are active per token (2 of 8 experts). DeepSeek-R1 has 671B total but only ~37B active. The model’s name says 47B, but it runs like a 13B.

Look for keywords: “Mixture of Experts,” “MoE,” “8x7B” (the “x” is a giveaway), or in config.json you’ll see fields like num_experts and num_experts_per_tok. If the card reports both “total parameters” and “active parameters,” it’s MoE. Memory requirements are based on total parameters (all experts must be loaded), but inference speed relates to active parameters.

Every transformer model uses an attention mechanism with queries (Q), keys (K), and values (V). The attention type determines how these are organized, affecting speed and memory during long-context inference.

Multi-Head Attention (MHA): Each attention head has its own Q, K, V projections. The original Transformer design. Full quality but highest memory for KV-cache.

Grouped-Query Attention (GQA): Multiple query heads share a smaller set of K/V heads. Llama 3, Gemma 2, and Mistral use this. Reduces KV-cache memory significantly with minimal quality loss.

Multi-Query Attention (MQA): All query heads share a single K/V head. Maximum speed, but slightly lower quality. Used in some older models like Falcon.

In config.json, compare num_attention_heads (query heads) with num_key_value_heads (KV heads). If they’re equal, it’s MHA. If KV is smaller (e.g., 32 query heads, 8 KV heads), it’s GQA. If KV is 1, it’s MQA. Most modern LLMs use GQA as the sweet spot.

Context length (or context window) is the maximum number of tokens the model can process in a single input+output. Common values: 2K–4K (older models), 8K (GPT-3.5 era), 32K–128K (modern LLMs like Llama 3.1, Gemma 2), 200K–1M+ (Gemini, Claude). A token is roughly 3/4 of a word in English, so 128K tokens ≈ 96,000 words ≈ a full novel.

Longer context means you can feed in more documents, longer conversations, or entire codebases. But: memory usage grows with context length (the KV-cache), quality can degrade at the edges of very long contexts (the “lost in the middle” problem), and cost scales linearly with input tokens. A model that advertises 128K context may perform great at 32K but poorly at 120K. Look for the card’s stated “effective context” or needle-in-a-haystack test results.

hidden_size × num_hidden_layers = the model’s capacity. More layers and wider layers = more parameters.

num_attention_heads vs num_key_value_heads = attention type (equal = MHA, different = GQA).

max_position_embeddings = the maximum context length. 131072 = 128K tokens.

rope_theta = the rotary position encoding base. Higher values (500K+) indicate models trained for long context. Lower values (10K) suggest shorter effective context.

VRAM (GB) ≈ Parameters (B) × Bytes per Parameter

FP32: 8B × 4 = 32GB
FP16/BF16: 8B × 2 = 16GB
INT8: 8B × 1 = 8GB
INT4 (Q4): 8B × 0.5 = 4GB

This is weights only. Add 20–30% for KV-cache, activations, and framework overhead. So an 8B model at FP16 realistically needs ~20GB, and at INT4 needs ~5–6GB. A 70B model at INT4 needs ~40GB — two 24GB GPUs or one 48GB GPU.

Bigger is generally better, but the relationship is logarithmic, not linear. Going from 7B to 70B (10x more parameters) doesn’t give you 10x better results — it might give you 15–25% improvement on benchmarks. Meanwhile, a well-fine-tuned 7B model can outperform a generic 70B model on a specific task. The current generation of “small” models (Llama 3.2 3B, Phi-4 14B, Gemma 3 4B) often match or beat previous-generation large models.

When choosing between model sizes, compare: benchmarks per GB of VRAM (efficiency), not raw benchmark scores. A 7B model scoring 75% on MMLU while needing 5GB is often more practical than a 70B model scoring 85% while needing 40GB. The question isn’t “which is smarter?” but “which gives me the best result I can actually run?”

1. How many parameters? Check if total vs. active differs (MoE).

2. Dense or MoE? Look for “8x7B” naming or num_experts in config.json.

3. What’s the context length? Check max_position_embeddings and match to your use case.

4. Will it fit my hardware? Params × bytes-per-param + 25% overhead.

5. What attention type? GQA = modern and efficient. MHA = older, more memory-hungry.

Architecture specs only matter in relation to your constraints. A chatbot for customer support might need a 7B model with 8K context. A code assistant might need a 32B model with 128K context. A document analysis pipeline might need a 70B model with 32K context. Let your use case drive the spec requirements, not the other way around.