[Paper Review] Understanding Black-box Predictions via Influence Functions

Pang Wei Koh, Percy Liang|arXiv (Cornell University)|Mar 14, 2017

Adversarial Robustness in Machine Learning38 references1,189 citations

TL;DR

The paper reuses influence functions to trace a model’s predictions back to training data, offering a scalable way to identify training points most responsible for a given prediction, and demonstrates applications in understanding, debugging, data cleaning, and adversarial training.

ABSTRACT

How can we explain the predictions of a black-box model? In this paper, we use influence functions -- a classic technique from robust statistics -- to trace a model's prediction through the learning algorithm and back to its training data, thereby identifying training points most responsible for a given prediction. To scale up influence functions to modern machine learning settings, we develop a simple, efficient implementation that requires only oracle access to gradients and Hessian-vector products. We show that even on non-convex and non-differentiable models where the theory breaks down, approximations to influence functions can still provide valuable information. On linear models and convolutional neural networks, we demonstrate that influence functions are useful for multiple purposes: understanding model behavior, debugging models, detecting dataset errors, and even creating visually-indistinguishable training-set attacks.

Motivation & Objective

• Motivate explainability of black-box models beyond prediction accuracy.
• Propose a method to attribute predictions to training data via influence functions.
• Enable practical computation of influence in modern models despite non-convexity and non-differentiability.
• Show diverse applications: model understanding, debugging, data quality, and adversarial training.

Proposed method

• Formulate the influence of upweighting or perturbing a training example on model parameters via Hessian-inverse times gradient of the loss.
• Derive closed-form expressions for the influence on a test loss and for perturbations of training inputs or labels.
• Use efficient Hessian-vector products and either conjugate-gradient or stochastic estimation to compute H^{-1} gradients without explicit Hessian inversion.
• Handle non-convex and non-differentiable settings with damping and smoothing techniques, validating accuracy against leave-one-out retraining.
• Provide practical recipe for computing influence through automatic differentiation frameworks.

Experimental results

Research questions

• RQ1 Can influence functions accurately identify training points most responsible for a given prediction in deep networks?
• RQ2 How can influence functions be efficiently computed at scale for modern models?
• RQ3 Do influence functions remain informative when convexity/differentiability assumptions fail?
• RQ4 Can influence functions be used for debugging, data quality checks, and adversarial training data?
• RQ5 What insights about model behavior arise when comparing different models using influence measures?

Key findings

• Influence functions closely approximate leave-one-out retraining changes in predictions for logistic regression and CNNs on MNIST and similar tasks.
• Efficiently computing influence via Hessian-vector products enables ranking training points by influence in O(np) with additional small-sample estimators; stochastic and CG methods provide practical speedups.
• Influence helps distinguish how models rely on training data versus mere proximity in input space, outperforming nearest-neighbor intuition in explaining predictions.
• Smoothing non-differentiable losses (e.g., hinge) yields accurate influence estimates, while non-convexity can be mitigated with damping.
• Training-set perturbations guided by influence functions can craft visually indistinguishable adversarial training examples that flip test predictions, highlighting security considerations.
• Influence functions successfully identify domain-mismatch causes and mislabeled data, aiding debugging and data cleaning.