Traditional software monitoring tracks uptime, error rates, and response times. For LLM systems, the server can return 200 OK while delivering a completely wrong, hallucinated, or harmful response. Your HTTP metrics will show green while your users are getting garbage. You need a fundamentally different monitoring approach.

Production LLM observability requires monitoring five dimensions simultaneously:

1. Cost: How much are you spending per query, per feature, per user?
2. Latency: How fast are responses? Where are the bottlenecks?
3. Quality: Are responses actually good? Is quality stable over time?
4. Safety: Are harmful outputs reaching users?
5. Hallucination: Is the system fabricating information?

The building block of LLM observability is the trace — a complete record of what happened for a single request. A trace captures:

• Input: The user query and system prompt
• Retrieval: Which documents were fetched (for RAG)
• LLM call: Model, tokens in/out, latency, cost
• Output: The generated response
• Post-processing: Guardrail checks, formatting

Tools like Langfuse, Phoenix, and LangSmith capture traces automatically with OpenTelemetry-compatible instrumentation.

LLM costs are unpredictable and can spike without warning. A runaway agent loop can burn $1,000 in minutes. A prompt change that adds 500 tokens can double your monthly bill. Without cost tracking, you discover budget overruns from your finance team, not your dashboard.

• Cost per query: Average and p95 cost per user request
• Cost per model: Breakdown by GPT-4o vs Claude vs Haiku
• Cost per feature: Which product features are most expensive?
• Cost per user: Are power users driving disproportionate costs?
• Daily/weekly budget: Set limits and alert when approaching

Break down costs by the dimensions that matter for your business. A typical attribution model:

• By model: 60% GPT-4o, 25% embeddings, 15% Haiku
• By pipeline stage: 40% generation, 35% retrieval, 25% guardrails
• By feature: 50% chat, 30% search, 20% summarization

This tells you where to optimize. If embeddings are 25% of cost, switching to a cheaper embedding model has high ROI.

• End-to-end latency: Total time from user query to response delivered
• Time to first token (TTFT): How long before the user sees anything? Critical for streaming UX
• Percentiles: Track p50 (median), p95 (most users), and p99 (worst case). Averages hide tail latency problems
• Per-component breakdown: How much time in retrieval vs generation vs guardrails?

• Streaming: Stream tokens to the user as they’re generated. TTFT drops from 2s to 300ms even though total time is the same
• Parallel execution: Run retrieval and guardrail checks in parallel where possible
• Model selection: Use faster models (GPT-4o-mini, Haiku) for simple queries, stronger models for complex ones
• Caching: Cache embeddings, retrieval results, and even full responses for repeated queries
• Prompt optimization: Shorter prompts = fewer tokens = faster generation

Run an LLM judge on 5–10% of production responses to continuously score quality. Track multiple dimensions:

• Relevancy: Does the response address the user’s actual question?
• Helpfulness: Is it actually useful, or just technically correct but unhelpful?
• Coherence: Does it make logical sense, or contradict itself?
• Completeness: Does it cover all aspects of the question?

Plot these scores over time. A declining trend signals a problem before users complain.

Supplement automated scoring with implicit user feedback:

• Thumbs up/down: Direct quality signal (if your UI supports it)
• Regeneration rate: Users clicking “try again” indicates dissatisfaction
• Follow-up questions: Indicate the first answer was incomplete
• Session abandonment: User leaves without resolution

Learn to recognize these patterns in your quality dashboards:

• Sudden cliff: Quality drops sharply on a specific date → model update or prompt change
• Gradual decline: Quality slowly erodes over weeks → data drift or concept drift
• Time-of-day patterns: Quality worse during peak hours → load-related timeouts or truncation
• Topic-specific drops: Quality fine overall but terrible for one topic → missing or outdated documents in RAG

• PII leakage: Names, emails, SSNs, credit card numbers appearing in responses
• Toxic content: Hate speech, harassment, explicit material, violent content
• Policy violations: Responses that violate your organization’s usage policies
• Prompt injection success: Users who successfully manipulated the model
• Jailbreak attempts: Users trying to bypass safety guardrails (even if unsuccessful — track the attempts)

• Real-time (pre-response): Block harmful outputs before they reach users. Adds 100–500ms latency but prevents harm. Essential for safety-critical applications
• Batch (post-response): Analyze responses after delivery. No latency impact but harm may have already occurred. Good for trend analysis and improving guardrails

Use both: Real-time for critical safety checks, batch for comprehensive analysis and guardrail improvement.

• Safety violation rate: % of responses flagged by safety classifiers
• PII leak rate: % of responses containing detected PII
• Prompt injection attempt rate: % of inputs classified as injection attempts
• Jailbreak success rate: % of injection attempts that bypassed guardrails
• False positive rate: % of safe responses incorrectly blocked (impacts UX)

• Source verification (RAG): Compare every claim in the response against the retrieved documents. Flag claims not supported by any source
• Self-consistency: Generate multiple responses to the same query. If they disagree on key facts, flag for review
• Confidence scoring: Low token-level probabilities correlate with hallucination. Models are less “confident” when fabricating
• LLM-as-judge: Ask a judge model: “Are all claims in this response supported by the provided context?”

• Hallucination rate: % of responses containing at least one unsupported claim
• Severity classification: Minor (wrong date) vs major (wrong medical advice) vs critical (fabricated citation)
• Category tracking: Fabricated facts, wrong attributions, invented citations, contradictions with source

Track these over time. A rising hallucination rate signals model degradation, data quality issues, or retrieval problems.

• Threshold alerts: Metric crosses a predefined boundary (e.g., cost > $500/day)
• Anomaly alerts: Metric deviates from its normal pattern (e.g., 2σ from 7-day rolling average)
• Rate-of-change alerts: Metric changing faster than expected (e.g., quality dropping 1%/hour)
• Composite alerts: Multiple metrics degrading simultaneously (strongest signal of a real problem)

Prefer anomaly detection over fixed thresholds when possible — it adapts to natural variation and seasonal patterns.

1. Detect: Alert fires automatically
2. Triage: Is this a real issue? Check the dashboard for context
3. Diagnose: Which pillar is affected? Examine traces for the affected time window
4. Mitigate: Rollback to previous config, disable the affected feature, or add an emergency guardrail
5. Fix: Root cause analysis and permanent fix
6. Learn: Add the failure case to your eval dataset so it’s caught automatically next time

Start with the 5 pillars as top-level traffic-light indicators. Green/yellow/red for each metric. Then add drill-down capability:

• Time-series plots: Spot trends and anomalies over days/weeks
• Per-feature breakdown: Which features need attention?
• Recent incidents: What went wrong recently and was it resolved?
• Eval suite status: Last CI/CD eval run results and trends
• Cost projection: At current rate, what’s the monthly bill?