[Paper Review] Representer Point Selection for Explaining Deep Neural Networks

Chih‐Kuan Yeh, Joon Sik Kim+2 more|arXiv (Cornell University)|Nov 23, 2018

Adversarial Robustness in Machine LearningComputer Science77 citations

TL;DR

The paper introduces representer points: training samples with positive or negative representer values that linearly decompose a neural network’s pre-activation outputs, enabling scalable, real-time explanations of predictions.

ABSTRACT

We propose to explain the predictions of a deep neural network, by pointing to the set of what we call representer points in the training set, for a given test point prediction. Specifically, we show that we can decompose the pre-activation prediction of a neural network into a linear combination of activations of training points, with the weights corresponding to what we call representer values, which thus capture the importance of that training point on the learned parameters of the network. But it provides a deeper understanding of the network than simply training point influence: with positive representer values corresponding to excitatory training points, and negative values corresponding to inhibitory points, which as we show provides considerably more insight. Our method is also much more scalable, allowing for real-time feedback in a manner not feasible with influence functions.

Motivation & Objective

• Explain neural network predictions by attributing them to training points through a representer decomposition.
• Provide a scalable alternative to influence functions for identifying influential training samples.
• Differentiate excitatory (positive) and inhibitory (negative) representer points to yield deeper model insights.
• Demonstrate practical utility in dataset debugging, misclassification understanding, and sensitivity analysis.

Proposed method

• Assume a neural network with a two-part parameterization: Phi(x, Theta)=Theta1 f, where f=Phi2(x, Theta2).
• Derive a representer decomposition: Phi(x_t, Theta*) = sum_i alpha_i k(x_t, x_i, alpha_i) with alpha_i = -(1/(2 lambda n)) * dL(x_i, y_i, Theta)/dPhi(x_i, Theta) and k(x_t, x_i, alpha_i) = alpha_i f_i^T f_t.
• Impose L2 regularization on Theta1 to ensure the representer form holds at stationary points.
• Propose practical training (Eq. 3) to obtain Theta*, enabling computation of representer values for any test point.
• Extend to generating representer points for a given pre-trained model by solving a convex problem with a suitable loss (softmax or ReLU) that keeps Phi(x_i, Theta) aligned with a given model.
• Discuss implementation details and practical considerations for real-time explanations and interpretability.

Experimental results

Research questions

• RQ1 Can a deep neural network prediction be expressed as a linear combination of training-point activations with trainable weights (representer values)?
• RQ2 Do positive representer values identify excitatory training points and negative values identify inhibitory points that illuminate the model’s behavior for a given test instance?
• RQ3 Is the representer-based explanation scalable and more informative than influence functions for identifying influential training samples?
• RQ4 Can representer values assist in dataset debugging, misclassification analysis, and sensitivity decomposition?
• RQ5 Can the framework be applied to pre-trained models and extended to diverse activations (softmax, ReLU) with suitable convex losses?

Key findings

• Pre-activation predictions can be decomposed into a weighted sum over training activations, with weights (representer values) indicating training-point influence.
• Positive representer values correspond to excitatory training points that support the prediction, while negative values correspond to inhibitory points that suppress it.
• The representer approach is more scalable and enables near real-time explanations compared to influence functions, with favorable computational costs.
• Representer points enable clearer visualization of why a model made or did not make a particular prediction, and can reveal dataset issues implicated in misclassifications.
• Experiments on CIFAR-10 and AwA demonstrate meaningful, interpretable excitatory/inhibitory examples and robust qualitative insights beyond traditional methods.