Subject
12 entries
Interpretability
Bookmarks
Inspectus: LLM Analytics and Visualization
Inspectus is labml.ai's tool for visualizing LLM internals — attention maps, token distributions, and model analytics rendered as interactive visualizations in Jupyter notebooks. Makes transformer attention patterns inspectable without custom code.
Simple probes can catch sleeper agents
Anthropic research showing that simple linear probes trained on internal activations can reliably detect 'sleeper agent' backdoors in LLMs — models trained to behave well normally but activate malicious behavior on a trigger. Evidence that activation-space representations of deceptive intent are linearly separable.
Probing Vision Transformers
A research repository probing the internal representations and attention mechanisms of Vision Transformers (ViT, DeiT, DINO). Notable finding: self-supervised DINO produces more salient attention maps than supervised models, suggesting better spatial semantics from self-supervision.
Explaining Transformer Model Predictions
A practical comparison of SHAP, Transformers Interpret, and Ferret for explaining Hugging Face transformer predictions. Key takeaway: different methods give different results for the same prediction — all require careful interpretation.
Interpretable Machine Learning
Christoph Molnar's free online book covering the theory and practice of interpretable machine learning — from inherently interpretable models (decision trees, linear regression) to post-hoc methods (SHAP, LIME, counterfactuals). The standard reference for understanding and explaining ML model behavior.
Shapash: Making Machine Learning Models Transparent
Shapash is MAIF's Python library for making ML models interpretable to non-technical stakeholders — wrapping SHAP and LIME with better visualizations and business-friendly explanations. Targets the gap between data scientists and decision-makers.
FACET: Human-Explainable AI
FACET is BCG Gamma's Python library for human-explainable AI — extending SHAP with interaction effects and redundancy-aware feature importance, plus simulation tools for model-based what-if analysis. More sophisticated than vanilla SHAP for understanding feature relationships.
SHAP: SHapley Additive exPlanations
SHAP (SHapley Additive exPlanations) is the standard Python library for explaining individual predictions from any ML model using game-theoretic Shapley values. It works across tree models, deep neural networks, and linear models, and produces both local and global interpretability.
ELI5 — sklearn Explainability Module
ELI5's sklearn module provides model explanation tools for scikit-learn estimators — feature importance, prediction decomposition, and permutation-based importance across linear models, tree ensembles, and SVMs. The explainability companion for sklearn workflows.
Visualizing Representations: Deep Learning and Human Beings
Christopher Olah's essay on visualizing what neural networks actually learn — using dimensionality reduction to show how deep networks transform data into progressively more separable representations. One of the most important early pieces on deep learning interpretability.
Interpreting Random Forests
A deep dive into techniques for interpreting random forest models — going beyond accuracy to understand what the model has learned. Particularly useful because random forests were the dominant ensemble method before gradient boosting took over, and interpretability was the main complaint against them.
Variable Importance in Neural Networks
R-bloggers post on measuring variable importance in neural networks — techniques like the Garson algorithm and Olden's method for attributing prediction contributions to input features. An early attempt at neural network interpretability before SHAP, LIME, and modern explainability tools existed.