LLMs are trained on public internet data with a knowledge cutoff date. They don’t know about your company’s products, your internal policies, your customer data, or anything that happened after training ended.

Ask GPT-4 about your company’s Q3 earnings, your return policy, or your latest product release — it will either hallucinate an answer or admit it doesn’t know.

For most enterprise AI products, the value comes from answering questions about your specific data. A customer support bot needs your knowledge base. A legal assistant needs your contracts. An internal tool needs your documentation.

Fine-tuning bakes knowledge into the model’s weights. It’s expensive, slow to update, and the model can still hallucinate. You can’t easily verify where an answer came from.

RAG (Retrieval-Augmented Generation) keeps knowledge external. At query time, the system retrieves relevant documents and includes them in the prompt. The LLM generates an answer grounded in those documents.

RAG advantages:
• Always current: Update the knowledge base, and answers change immediately
• Citable: Every answer can point to its source documents
• Cheaper: No retraining required when data changes
• Controllable: You decide exactly what knowledge is available

RAG is the default architecture for enterprise LLM products in 2025–2026. Fine-tuning is used for behavior and style; RAG is used for knowledge.

This happens before any user query. You prepare your knowledge base for fast retrieval:

1. Load documents.
Ingest raw data from all sources: PDFs, web pages, databases, Confluence, Notion, Slack, email archives, support tickets. Both batch backfills (historical data) and incremental updates (new data as it arrives).

2. Chunk documents.
Split large documents into smaller pieces (chunks). A 50-page PDF becomes hundreds of 512-token chunks. Each chunk should be a self-contained unit of information.

3. Embed chunks.
Convert each text chunk into a numerical vector (embedding) that captures its meaning. Similar texts produce similar vectors.

4. Store in vector database.
Save the embeddings in a specialized database (Pinecone, Weaviate, pgvector, Qdrant) that supports fast similarity search across millions of vectors.

This happens for every user query in real time:

5. Embed the query.
Convert the user’s question into a vector using the same embedding model.

6. Search the vector database.
Find the top-K chunks whose vectors are most similar to the query vector. Typically K = 5–20.

7. Re-rank results.
Use a cross-encoder model to re-score the retrieved chunks for relevance. This boosts precision by 18–42% over vector search alone.

8. Build the prompt.
Combine the system prompt + retrieved chunks + user query into a single prompt for the LLM.

9. Generate the answer.
The LLM reads the retrieved context and generates an answer grounded in those documents.

10. Cite sources.
Include references to the source documents so users can verify the answer.

How you split documents into chunks has an outsized impact on retrieval quality:

Fixed-size chunking (recommended start):
Split at 512 tokens with 10–15% overlap between chunks. Simple, predictable, and surprisingly effective. Production data from 2026 shows this outperforms more complex approaches in most cases.

Semantic chunking:
Split at natural boundaries (paragraphs, sections, topic shifts). Sounds better in theory but creates 3–5x more fragments, increasing embedding costs and retrieval noise. Use only when fixed-size demonstrably fails.

Document-aware chunking:
Respect document structure (headers, tables, code blocks). Important for structured content like technical documentation or legal contracts where splitting mid-table destroys meaning.

The overlap matters: Without overlap, information that spans a chunk boundary is lost. A 10–15% overlap ensures continuity.

What embeddings do:
Convert text into vectors (lists of numbers) that capture semantic meaning. “How do I return a product?” and “What is your refund policy?” produce similar vectors even though they share few words.

Model selection:
• text-embedding-3-small: Best cost/quality balance for most use cases ($0.02 per million tokens)
• text-embedding-3-large: Higher quality, 9x more expensive
• Open-source alternatives (e.g., BGE, E5): Free to run but require hosting infrastructure

Cost at scale:
Embedding 1 million documents at 512 tokens each costs $10–$90 depending on the model. Re-embedding when you change models costs the same again. Choose carefully upfront.

Store metadata alongside each chunk: source URL, document title, author, last updated date, section heading, document type. This enables filtering (“only search HR policies”), temporal weighting (“prefer recent documents”), and source citation in answers.

How it works: Convert the query to a vector, find the nearest vectors in the database using approximate nearest-neighbor (ANN) algorithms like HNSW.

Strengths: Captures semantic meaning. “How do I cancel my subscription?” matches a document about “account termination procedures” even without shared keywords.

Weaknesses: Misses exact keyword matches. Searching for “error code E-4021” might return documents about errors in general rather than the specific code. Also struggles with ambiguous queries (“cellular” could mean biology or phone plans).

How it works: Traditional text search based on word frequency and document length. The same technology behind search engines.

Strengths: Exact matches. “Error code E-4021” finds exactly the document containing that code. Fast and well-understood.

Weaknesses: No semantic understanding. “How do I cancel?” won’t match “account termination” because the words are different.

Combine both. Run vector search and keyword search in parallel, then merge results using Reciprocal Rank Fusion (RRF). Documents that rank highly in both searches get boosted to the top.

Hybrid search is now the production standard for RAG systems. It captures both semantic meaning and exact matches, covering each approach’s blind spots.

After initial retrieval, pass the top 20–50 results through a cross-encoder re-ranker. Unlike embedding models (which encode query and document separately), cross-encoders process the query and document together, producing more accurate relevance scores.

Re-ranking boosts precision by 18–42% in production systems. It’s slower (can’t search the full database this way) but dramatically improves the final set of documents sent to the LLM.

Apply time-aware weighting so recent documents outrank stale ones. Critical for knowledge bases where policies change, products update, and old information becomes misleading.

1. Wrong documents retrieved.
The query “cellular data plan” retrieves biology documents about cellular structures. Ambiguous queries are the most common retrieval failure. Mitigation: hybrid search, query expansion, metadata filtering.

2. Relevant documents not in the knowledge base.
The user asks about a product that was launched last week, but the knowledge base hasn’t been updated. Mitigation: automated ingestion pipelines with freshness monitoring.

3. Stale documents outranking current ones.
An outdated return policy from 2023 ranks higher than the current 2026 policy because it has more keyword matches. Mitigation: temporal filtering, document versioning, explicit deprecation.

4. Noisy retrieval.
The top-K results include 3 relevant and 7 irrelevant chunks. The irrelevant chunks confuse the LLM and trigger hallucinations. Mitigation: re-ranking, stricter relevance thresholds, fewer but higher-quality chunks.

5. Hallucination despite context.
The LLM ignores the retrieved documents and generates information from its training data. The answer sounds authoritative but isn’t grounded in your knowledge base. Mitigation: strong grounding instructions in the system prompt, faithfulness evaluation.

6. Synthesizing across contradictory sources.
Two retrieved documents give conflicting information (old policy vs. new policy). The LLM blends them into a confidently wrong answer. Mitigation: document versioning, conflict detection, preferring the most recent source.

7. Missing the answer in the context.
The relevant information is in the retrieved chunks, but the LLM fails to find it — especially in long contexts. LLMs have a “lost in the middle” problem where they attend more to the beginning and end of the context. Mitigation: put the most relevant chunks first, limit context length.

Recall@K: Of all relevant documents, how many appear in the top K results? Target: >85% at K=10 for production systems.

Precision@K: Of the top K retrieved documents, how many are actually relevant? Higher precision means less noise for the LLM.

Mean Reciprocal Rank (MRR): How early does the first relevant result appear? Important because LLMs pay more attention to earlier context (positional bias).

Context relevance: Semantic alignment between the query and retrieved passages. Measured using embedding similarity or cross-encoder scores.

Faithfulness / Groundedness: Does the answer stay grounded in the retrieved context? Or does it introduce information not present in the documents? This is the hallucination metric for RAG.

Answer relevancy: Does the answer actually address the user’s question? A grounded but off-topic answer is still a failure.

Completeness: Does the answer cover all relevant aspects from the retrieved context? Partial answers frustrate users.

Executive layer: Manual accuracy checks on 200–1,000 representative questions. Simple, defensible KPIs. Inter-rater agreement (Cohen’s kappa).

Team layer: Standardized frameworks like RAGAS for reproducible, automated evaluation across retrieval and generation dimensions.

Developer layer: Fine-grained telemetry for debugging. Per-query retrieval scores, chunk-level relevance, latency breakdowns, hallucination source tracing.

The knowledge base needs two ingestion lanes:

Batch backfill: Initial load of all historical documents. Run once at setup, then periodically for full re-indexing (e.g., when you change the embedding model or chunking strategy).

Incremental updates: New and updated documents ingested continuously. A new support article published at 2pm should be retrievable by 2:15pm. This requires automated pipelines that detect changes, re-chunk, re-embed, and update the vector database.

Deletion and deprecation: When documents are outdated or removed, the corresponding chunks must be deleted from the vector database. Stale chunks are a common source of wrong answers.

Source quality matters most. RAG can’t fix bad source documents. If your knowledge base contains contradictory, outdated, or poorly written content, the AI will faithfully reproduce those problems.

Common data quality issues:
• Duplicate documents (same info, different versions) → conflicting answers
• Outdated content not marked as deprecated → stale answers
• Poorly structured documents (no headings, no sections) → bad chunking
• Missing metadata (no dates, no authors) → can’t filter or prioritize
• Inconsistent terminology across documents → retrieval gaps

The PM’s role: Champion a knowledge base hygiene program. Regular audits, clear ownership of content areas, deprecation workflows, and quality standards for new content. This is often the highest-leverage investment in RAG quality.

1. Scope the knowledge base.
What data sources are included? What’s excluded? Start narrow (one document collection) and expand. Every new source adds complexity and potential failure modes.

2. Define freshness requirements.
How quickly must new information be available? Real-time (minutes), near-real-time (hours), or batch (daily)? Freshness requirements drive infrastructure cost.

3. Set the citation standard.
Must every answer cite its sources? Can the AI say “I don’t know”? How are citations displayed? Source citation is the primary trust mechanism in RAG products.

4. Define the “I don’t know” behavior.
When no relevant documents are found, the AI should say so rather than hallucinate. This requires a retrieval confidence threshold: below it, the system admits uncertainty rather than guessing.

Level 1: Basic RAG
Vector search, fixed chunking, single data source. Good for internal tools and proofs of concept. 2–4 weeks to build.

Level 2: Production RAG
Hybrid search, re-ranking, multiple data sources, automated ingestion, source citations. Good for customer-facing products. 2–3 months to build.

Level 3: Advanced RAG
Agentic RAG (the AI decides what to search and when), multi-step retrieval, query decomposition, graph-based knowledge structures. For complex domains with interconnected information. 4–6+ months.

Level 4: Enterprise RAG
Multi-tenant access control, compliance audit trails, real-time ingestion, cross-language retrieval, feedback-driven re-indexing. For regulated industries at scale. Ongoing investment.