monitoring

LLM Evaluation & Observability

Measure what matters — benchmarks, judges, guardrails, and production monitoring for AI systems that actually work.
Co-Created by Kiran Shirol and Claude
TopicsBenchmarksLLM-as-JudgeGuardrailsObservabilityEval Pipelines
home Learning Portal play_arrow Start Learning dictionary Glossary summarize Key Insights12 chapters · 4 sections
Section 1

Foundations — How to Measure AI

Why evaluation matters, benchmarks, automated judges, and human preference.
1
priority_high
Why Evaluation Matters
The “works on my laptop” problem, silent failures, and why vibes-based eval fails at scale.
arrow_forward Learn
2
leaderboard
Benchmarks: The Scoreboard
MMLU, HumanEval, SWE-bench, GPQA — what they measure and why they saturate.
arrow_forward Learn
3
gavel
LLM-as-Judge
Using LLMs to evaluate LLMs — 80-90% human agreement at 5000x lower cost.
arrow_forward Learn
Section 2

Evaluating AI Systems

RAG, agents, human evaluation, and building systematic eval pipelines.
4
search
Evaluating RAG Systems
RAGAS metrics: faithfulness, answer relevancy, context precision, and groundedness.
arrow_forward Learn
5
smart_toy
Evaluating Agents
Task completion, tool use accuracy, trajectory evaluation, and SWE-bench for agents.
arrow_forward Learn
6
group
Human Evaluation
Chatbot Arena, preference ranking, annotation guidelines, and when humans are essential.
arrow_forward Learn
7
account_tree
Building an Eval Pipeline
From ad-hoc to systematic: datasets, metrics, CI/CD gates, and regression testing.
arrow_forward Learn
8
handyman
The Eval Tools Landscape
RAGAS, DeepEval, Braintrust, LangSmith, Arize Phoenix, Langfuse — when to use which.
arrow_forward Learn
Section 3

Production — Observability & Guardrails

Monitoring, safety, drift detection, and keeping AI systems healthy in production.
9
monitoring
Production Observability
5 pillars: cost tracking, latency profiling, quality monitoring, and hallucination detection.
arrow_forward Learn
10
shield
Guardrails & Safety
Input/output guardrails, PII detection, prompt injection defense, and content filtering.
arrow_forward Learn
11
troubleshoot
Drift, Debugging & Alerts
Quality drift detection, model update regressions, root cause analysis, and incident response.
arrow_forward Learn
Section 4

Mastery — The Eval-First Mindset

Building evaluation into your culture, not bolting it on after.
12
star
The Eval-First Mindset
Eval-driven development, the eval checklist, building a culture, and what to measure at each stage.
arrow_forward Learn
explore

Explore Related Courses

search
RAG
Retrieval-Augmented Generation
psychology
Agentic AI
Planning, Memory & Tool Use
shield
AI Security
Threats & Defenses
code
AI-Assisted Coding
From Completion to Agents