Interpretability and Explainable AI

Explanatory targets

• Predictor and behavior - Inspect the scoring rule, explain one output, or measure predictive performance. Removal-based explanations distinguish output contributions from loss-based importance; these targets need different comparisons. Covert et al. 2021.
• Training data - Influence methods estimate how training observations affect predictions or loss. Koh and Liang approximate parameter responses; deletion-and-retraining is a separate validation target. Koh and Liang 2017.
• Representation and computation - Probes test decodable information; causal abstractions test proposed computations through aligned interventions. Accessible information alone does not establish its use. Hewitt and Liang 2019, Geiger et al. 2021.
• Decision changes - Recourse concerns feasible actions and their downstream effects. Intervening on model activations tests the program; intervening on people or physical systems requires assumptions about the world. Karimi et al. 2020.

Interpretable models

• Sparse scoring systems - Learn small integer coefficients with sparsity and application-specific constraints. The score is the predictor itself. Ustun and Rudin 2019.
• Rule lists - Ordered conditions followed by a default prediction. CORELS certifies optimal regularized training risk over its supplied rule candidates when optimization completes. Angelino et al. 2018.
• Additive models and interactions - GAMs sum feature functions on a link/score scale; GA²M adds pairwise terms. EBMs learn them by boosting. Additivity on that scale need not imply additive probability changes. Lou et al. 2013; docs.
• Neural additive models - Jointly train one neural network per input and sum their outputs on the score/link scale. The original architecture exposes feature functions without explicit cross-feature interaction terms. Agarwal et al. 2021.
• RETAIN - Uses reverse-time, two-level attention and computes visit/variable contributions with attention held fixed. Interpretation relies on this architecture's contribution calculation. Choi et al. 2016; code.

Feature effects and global importance

• Removal-based explanations - Covert et al. connect SHAP, LIME, permutation tests and other methods through three choices: how features are removed, which behavior is measured, and how removal effects are summarized. Covert et al. 2021.
• Partial dependence (PDP) - Averages model predictions while varying selected inputs and retaining other observed values. Correlated inputs can produce unrealistic combinations. docs.
• Individual conditional expectation (ICE) - Plots a separate prediction curve for each observation, exposing heterogeneity hidden by average partial dependence. Goldstein et al. 2013.
• Accumulated local effects (ALE) - Accumulates locally averaged prediction differences, reducing extrapolation caused by dependent inputs. Feature effects describe the fitted model, not causal effects. Apley and Zhu 2016.
• Permutation importance - Measures a fitted model's score change after shuffling one input. Results depend on the model, score, evaluation data, and correlated substitutes. docs.
• Conditional subgroup importance - Forms interpretable subgroups with trees, then permutes within them to approximate conditional feature importance; also adapts PDPs. Molnar et al. 2023.
• Dependence and extrapolation - Breaking feature relationships can make importance and effect plots reflect behavior in sparsely observed regions. Hooker and Mentch analyze this failure and alternatives. Hooker et al. 2019.
• SAGE - Assigns Shapley values to features using their contribution to expected predictive loss reduction. This measures predictive power rather than averaging local prediction attributions. Covert et al. 2020.
• Model class reliance - Ranges over feature importance across a specified class of sufficiently accurate predictors. Different models can rely on different inputs; the class and performance tolerance define the range. Fisher et al. 2019.

Attribution and local explanations

Feature attribution

• SHAP - Decomposes an individual prediction into additive Shapley contributions. Aggregation summarizes contributions over chosen observations; it does not turn them into loss-based importance. Lundberg and Lee 2017; code.
• TreeSHAP - Computes tree-model attributions under specified assumptions about missing or dependent features. Background data and perturbation mode affect the explanation. Lundberg et al. 2018; docs.
• Shapley-value choices - The value function and treatment of absent features define the attribution problem; different constructions can disagree. Sundararajan and Najmi 2020.
• Duplicate-input example (constructed) - Let X2=X1 be a fair bit and f(x)=x1. At (1,1), conditional Shapley gives (0.25,0.25); marginal replacement gives (0.5,0). Game definitions change whether an unused but informative input receives credit. Sundararajan and Najmi 2020.
• Integrated Gradients - Integrates input gradients along a straight path from a baseline to the observation. Attributions sum to the output difference; the baseline is part of the explanation. Sundararajan et al. 2017.
• DeepLIFT - Propagates activation differences relative to reference activations to assign input contributions. Shrikumar et al. 2017; code.
• Layer-wise relevance propagation (LRP) - Redistributes a prediction backward through layers using relevance-propagation rules. The propagation rule determines the decomposition. Bach et al. 2015.
• Grad-CAM - Weights convolutional feature maps with target gradients to produce a coarse spatial attribution map. A heatmap's visual appeal does not establish faithfulness. Selvaraju et al. 2019.
• Attribution guarantees - IG completeness constrains a sum; Shapley axioms characterize an allocation for a specified game. Neither property alone identifies a unique internal computation. Sundararajan et al. 2017, Sundararajan and Najmi 2020.

Local surrogates and rules

• LIME - Fits an interpretable surrogate around one prediction using weighted perturbed samples. Sampling, neighborhood weights, and surrogate capacity shape the result. Ribeiro et al. 2016; code.
  • MeLIME - Extends LIME with data-distribution-aware sampling and alternative local interpretable models. Its changes address the perturbation and surrogate-fitting stages. Botari et al. 2020.
• Anchors - Searches for rules that preserve a prediction with high probability under a specified perturbation distribution. Precision depends on that distribution. Ribeiro et al. 2018; code.

Formal sufficient explanations

• Abductive explanations - Find input assignments sufficient to preserve a prediction for every completion in the encoded domain. The guarantee requires a model/domain encoding and a sound constraint solver. Ignatiev et al. 2018.
• Minimality and scope - Subset-minimal means no assignment can be dropped; minimum-cardinality means fewest assignments. Unlike Anchors' perturbation-distribution precision, formal sufficiency quantifies over all admissible completions. Ignatiev et al. 2018.

Counterfactual explanations and recourse

• Counterfactual explanations - Search for nearby input changes that produce a desired model output. A changed prediction alone does not establish an actionable intervention. Wachter et al. 2017.
• Diverse counterfactuals - DiCE balances proximity and diversity to offer several ways to change a prediction. Feature constraints specify permitted changes. Mothilal et al. 2020; code.
• Actionable recourse - Optimizes changes to actionable inputs for linear classifiers. Karimi et al. distinguish input edits from interventions with downstream causal effects. Ustun et al. 2018, Karimi et al. 2020.

Training-data attribution

• Influence functions - Approximate how upweighting a training example changes a test prediction or loss using derivatives of the fitted model. Deep-network approximations require validation. Koh and Liang 2017.
• TracIn - Estimates training-example influence using gradient inner products across saved training checkpoints. This differs from a single-checkpoint Hessian approximation. Pruthi et al. 2020.
• Datamodels - Predict a target example's model output from which training examples were included. Ilyas et al. fit these surrogates using many subset-trained models, enabling dataset counterfactuals; the training algorithm and subset distribution define the target. Ilyas et al. 2022.
• TRAK - Approximates training-data effects using randomly projected model gradients and a kernel-based calculation. Evaluated against retraining-based prediction changes across vision and language tasks; these scores remain approximations to dataset interventions. Park et al. 2023.

Concepts and representations

• TCAV - Learns concept directions from examples and measures prediction sensitivity through directional derivatives. Results depend on concept examples and the chosen representation layer. Kim et al. 2018.
• Concept bottleneck models - Predict annotated concepts before predicting the target; editing predicted concept values changes downstream predictions. These are interventions inside the model. Koh et al. 2020.
• Concept leakage - Soft concept outputs can encode task information beyond their labels, even with sequential training. Mahinpei et al. demonstrate this on synthetic and MNIST tasks; concept supervision does not ensure semantic isolation. Mahinpei et al. 2021.
• ProtoPNet - Predicts by matching image regions to learned prototypes associated with training-image patches. Prototype similarity forms part of the predictor, rather than a separate local surrogate. Chen et al. 2019.
• Probing with controls - Test what a representation makes decodable while checking what the probe itself can memorize. Decodability alone does not establish that the original model uses that information. Hewitt and Liang 2019.
• Visual representations - Feature visualization optimizes inputs for activation; Network Dissection measures unit alignment with labeled concepts (2017). Activation patterns and semantic alignment do not by themselves test causal use. Olah et al. 2017, Bau et al. 2017.

Mechanistic interpretability

• Causal abstraction - Aligns neural representations with variables in a proposed causal model and tests corresponding interchange interventions. This evaluates a specified computational hypothesis, not causal relations in the world. Geiger et al. 2021.

Candidate representations

• Superposition - Controlled toy models store sparse features in overlapping directions, yielding polysemantic neurons. This motivates sparse dictionaries; it does not establish that a learned dictionary recovers a real network's true features. Elhage et al. 2022.
• Sparse autoencoders - Reconstruct activations as sparse combinations of learned directions, seeking features distributed across neurons. Bricken et al. 2023, Gao et al. 2024.
• Feature absorption - A child concept can absorb a parent concept in an SAE, causing the parent latent to miss examples. Larger dictionaries or greater sparsity need not resolve this failure. Chanin et al. 2025.
• Native feature baselines - Feed-forward key/value features compete with SAEs and transcoders on the tested interpretability measures. Dictionary widths and available checkpoints constrain the comparison. Ye et al. 2025.
• SAE seed sensitivity - Different initializations on the same model/data yield different feature dictionaries. Overlap depends on architecture and matching; one fitted SAE is not a unique feature inventory. Paulo and Belrose 2025.
• Neurons versus transcoders - Many late-layer transcoder activations align with MLP neurons. Sparsified neurons need a learned linear output correction for competitive replacement KL divergence; transcoders better reconstruct outputs. Shabalin and Piotrowski 2026.
• Neuron comparison scope - Neurons match transcoders on top-activation sort tests but fare worse across broader activation quantiles; transcoders better detect fine-grained concepts such as bigrams. Comparisons, mainly English data. Shabalin and Piotrowski 2026.

Interventions and circuit discovery

• Activation patching - Replaces internal activations between runs to test their effect on model behavior. Zhang and Nanda show that corruption choices and evaluation metrics can change localization results. Zhang and Nanda 2024.
• Transformer circuit framework - Decomposes attention computation into query-key and output-value paths, including interactions across layers. The initial analysis studies small attention-only models. Elhage et al. 2021.
• Circuit discovery case study - Identifies attention-head roles in GPT-2 small's indirect-object identification task and tests faithfulness, completeness, and minimality. Findings concern that model and task. Wang et al. 2022.
• ACDC - Removes computational edges through activation-patching tests while seeking to retain a specified behavior. The recovered circuit depends on the dataset, intervention and metric. Conmy et al. 2023.
• Sparse Feature Circuits / SHIFT - Humans select task-irrelevant SAE circuit features for ablation, then retrain the classifier head. Marks et al. improve profession classification on gender-balanced biographies in their case study. Marks et al. 2025.
• Circuit tracing - Builds prompt-specific attribution graphs using transcoders that approximate model components. Reconstruction error and frozen attention limit the original method. Ameisen et al. 2025.
• Attention-path extensions - Traces feature interactions through attention query-key computations omitted by the original circuit-tracing method. Kamath et al. 2025.
• Neuron circuits - MLP activations with RelP attribution yield sparse circuits competitive with SAE circuits on tested tasks. Activation site and attribution rule both matter; sparsity does not establish monosemanticity. Arora et al. 2026.

Mechanism evaluation

• MIB: circuit localization - Four tasks and five models; attribution and mask-optimization methods lead on the benchmark's circuit metrics. Scores trade behavioral recovery against circuit size under specified ablations. Mueller et al. 2025.
• MIB: causal-variable localization - Supervised distributed alignment search leads; tested SAE features do not improve over neuron coordinates. This compares task-aligned intervention effects, not general feature readability. Mueller et al. 2025.
• Circuit-score misranking - Geng et al. find intervention scores favoring equally sized circuits with worse held-out behavioral agreement. Restoring intact-model signals reverses selected failures; mechanism identity remains unverified. Geng et al. 2026.
• SAEBench - Evaluates sparse autoencoders across eight metrics spanning interpretation, disentanglement, and downstream uses; reconstruction gains need not improve other scores. Karvonen et al. 2025; code.
• SAEBench reliability - Chanin's audit tests reseed noise, synthetic ground-truth agreement and training-trajectory discrimination. Targeted Probe Perturbation and Spurious Correlation Removal fail multiple tests at canonical settings. Chanin 2026.
• Mechanism validation protocol - Record checkpoint, task, intervention, reference behavior and metric; separate hypothesis-discovery examples from held-out tests. MIB supplies task datasets and intervention-based comparisons. Mueller et al. 2025.

Evaluation and comparative evidence

• Evaluation objectives - Distinguish application-grounded, human-grounded, and functionally grounded evaluation. A proxy score and usefulness to a person answer different questions. Doshi-Velez and Kim 2017.
• Human usefulness - Transparent, few-feature apartment-price models improved prediction simulation, but not advice-following; transparency sometimes impaired error correction. 3,800 participants; effects depended on presentation. Poursabzi-Sangdeh et al. 2018.
• Sources of disagreement - Separate explainer approximation noise, changes from model fitting, and different accurate models. Changing the target is distinct from estimating the same target unreliably. Sundararajan and Najmi 2020, Fisher et al. 2019.

Faithfulness, stability, and perturbations

• Faithfulness versus plausibility - An explanation's agreement with model behavior differs from whether a person finds it convincing. Jacovi and Goldberg distinguish these evaluation targets. Jacovi and Goldberg 2020.
• Saliency sanity checks - Randomize model parameters or training labels to test whether explanations depend on learned behavior. Some tested methods remained visually similar after randomization. Adebayo et al. 2018.
• Infidelity and sensitivity - Compare attributed changes with output changes under perturbations, and measure explanation variation. A constant explanation can minimize sensitivity without being informative. Yeh et al. 2019.
• Remove-and-retrain (ROAR) - Masks highly ranked inputs and retrains to assess retained predictive information. This tests a different question from perturbing a fixed predictor. Hooker et al. 2019.
• Out-of-distribution perturbations - Masking can make feature-importance scores and evaluation metrics reflect behavior outside the training distribution. Perturbation design is part of the evaluation. Hase et al. 2021.
• Adversarial explanations - Classifiers can behave differently on real and perturbed inputs, concealing biased predictions from LIME/SHAP in the studied attacks. Slack et al. 2019.
• Influence-estimation failures - Network depth, regularization, and test-point choice affect approximation accuracy in the investigated neural networks. Reported influence is not an exact retraining result. Basu et al. 2021.

Attention and language explanations

• Attention as explanation - Jain and Wallace find alternative attention weights with similar predictions; Wiegreffe and Pinter challenge the inference and propose additional tests (2019). The result depends on architecture and evaluation. Jain and Wallace 2019, Wiegreffe and Pinter 2019.
• Chain-of-thought faithfulness - Models can use prompt hints or biases without reporting them in their explanations. Chen et al. test reasoning models. Findings are conditional on the tested tasks and hints. Turpin et al. 2023, Chen et al. 2025.
• Interpretation pitfalls - Feature dependence, interactions, poor generalization, uncertainty, and unjustified causal readings can invalidate conclusions. Molnar et al. 2020; docs.

Software

• Interpretable models - InterpretML provides Explainable Boosting Machines and post-hoc methods; framework paper. Nori et al. 2019; code.
• Feature effects and attribution - scikit-learn for PDP/ICE and permutation importance; SHAP, LIME, ELI5 for model explanations. docs, code, code, docs.
• Neural-network attribution - Captum provides input, neuron, and layer attribution plus evaluation metrics for PyTorch; DeepLIFT supplies reference-based propagation. Kokhlikyan et al. 2020; docs, code.
• Explanation toolkits - AIX360 and Alibi collect multiple explanation methods; DiCE generates counterfactual examples. code, code, code.
• Mechanistic analysis - TransformerLens exposes activations and interventions; SAELens trains and analyzes sparse autoencoders; circuit-tracer constructs attribution graphs and supports interventions. code, code, code.
• Explanation evaluation - Quantus collects metrics for neural-network explanations; Metric assumptions and perturbation choices remain part of the comparison. Hedström et al. 2022; code.
• TSViz - Combines input/filter influence, filter clustering, and inverse optimization to inspect convolutional time-series models. Siddiqui et al. 2019.

Surveys and perspectives

• Interpretable Machine Learning - Molnar. Interpretable models, feature effects, attribution, examples, and limitations. Molnar 2025.
• Explanatory Model Analysis - Model diagnostics and local/global explanations, with R and Python examples. Biecek and Burzykowski 2021.
• Interpretable models versus post-hoc explanations - Rudin argues for directly interpretable predictors in high-stakes settings where suitable models can be built. Rudin 2018.
• Already Clean - Goodfire research essay argues that better visualizations and concept descriptions explain apparent polysemanticity. Its neuron examples motivate direct baselines; broad cleanliness claims remain preliminary. Cammarata 2026.

Applications

• Glucose forecasting - Applies RETAIN to longitudinal glucose prediction and examines its explanations. Evidence concerns the studied type-2 diabetes cohort. De Bois et al. 2020.

Related topics

• Feature selection
• Feature extraction
• Causal inference
• Neural networks
• Ensemble learning
• Models