Modern ML models — deep neural networks, gradient-boosted ensembles, LLMs — are black boxes. They make predictions, but we can’t easily understand why. This matters because: Trust — doctors won’t follow an AI diagnosis they can’t understand. Debugging — if you can’t explain why the model made a mistake, you can’t fix it. Fairness — you can’t detect bias if you can’t see what features drive decisions. Regulation — the EU AI Act requires explainability for high-risk AI; GDPR grants a “right to explanation” for automated decisions. Accountability — when an AI denies someone a loan, they deserve to know why. Two related concepts: interpretability (the model is inherently understandable) and explainability (a separate method explains a black-box model).

SHAP (Lundberg & Lee, 2017) uses Shapley values from cooperative game theory to assign each feature a contribution to the prediction. The Shapley value of a feature is the average marginal contribution of that feature across all possible feature combinations. Key properties: consistency (if a feature’s contribution increases, its SHAP value increases), local accuracy (SHAP values sum to the difference between the prediction and the average prediction), and missingness (missing features get zero attribution). SHAP provides both local explanations (why this specific prediction) and global explanations (which features matter most overall). Variants: TreeSHAP (fast for tree models), DeepSHAP (for neural networks), KernelSHAP (model-agnostic).

LIME (Ribeiro et al., 2016) explains individual predictions by creating a local approximation of the black-box model. For a given prediction: (1) generate perturbed samples around the input, (2) get the black-box model’s predictions for these samples, (3) fit a simple, interpretable model (linear regression) to these local predictions, (4) use the interpretable model’s coefficients as the explanation. LIME is model-agnostic (works with any model) and provides local explanations (specific to one prediction). Variants: LimeTabular (structured data), LimeText (NLP — highlights important words), LimeImage (computer vision — highlights important regions). Limitation: explanations can be inconsistent — running LIME twice on the same input may give different explanations (65–75% feature ranking overlap).

SHAP advantages: theoretically grounded (Shapley values), consistent explanations (same input → same explanation), provides both local and global explanations, fast for tree models (TreeSHAP). LIME advantages: faster for non-tree models, works well for text and images, simpler to understand conceptually, lighter-weight implementation. Choose SHAP when: you need consistent, reproducible explanations; you’re using tree-based models; you need global feature importance; regulatory compliance requires rigorous attribution. Choose LIME when: you need quick, one-off explanations; you’re explaining text or image models; you want intuitive, human-readable explanations; speed matters more than theoretical rigor.

Model cards (Mitchell et al., 2019, Google) are standardized documentation for ML models. They serve as a “nutrition label” for AI, communicating what the model does, how it was built, and its limitations. A model card includes: Model details (architecture, training data, version), Intended use (what it’s designed for and what it’s NOT designed for), Performance metrics (accuracy, disaggregated by demographic group), Limitations (known failure modes, edge cases), Ethical considerations (potential harms, bias analysis), and Training data (source, size, known biases). Model cards are required by the EU AI Act for high-risk AI systems and are considered best practice by Google, Hugging Face, and the ML community.

GDPR Article 22: individuals have the right not to be subject to decisions based solely on automated processing that significantly affect them. They can request “meaningful information about the logic involved.” EU AI Act: high-risk AI systems must be “sufficiently transparent to enable users to interpret the system’s output and use it appropriately.” Penalties: up to €35M or 7% of global turnover. US: no federal AI explainability law yet, but state laws are emerging (California SB-1001 requires disclosure of AI use). Equal Credit Opportunity Act: lenders must provide specific reasons for credit denials — “the algorithm said no” is not sufficient. The trend is clear: explainability is moving from best practice to legal requirement.

LLMs present unique explainability challenges: they have billions of parameters, process text in complex ways, and generate free-form output. Current approaches: Attention visualization — show which input tokens the model “attends to” when generating each output token. Intuitive but research shows attention weights don’t always correspond to importance. Chain-of-thought prompting — ask the model to “explain its reasoning step by step.” The explanation is human-readable but may not reflect the model’s actual computation (it’s a post-hoc rationalization). Mechanistic interpretability — reverse-engineer what individual neurons and circuits in the model do. Cutting-edge research (Anthropic, OpenAI) but not yet practical for most teams. RAG attribution — for retrieval-augmented generation, cite the source documents that informed the answer.

Step 1: Start with an interpretable model (logistic regression, decision tree, GAM). Only use a black box if the accuracy gain justifies the transparency loss. Step 2: For black-box models, add SHAP explanations. Use TreeSHAP for tree models, KernelSHAP for others. Generate both local (per-prediction) and global (feature importance) explanations. Step 3: Create a model card documenting the model, its performance (disaggregated), limitations, and ethical considerations. Step 4: For user-facing systems, provide explanations in plain language (“Your application was declined primarily because of insufficient credit history”). Step 5: For LLMs, use chain-of-thought + RAG attribution. Step 6: Log explanations alongside predictions for audit trails.