[Paper Review] Neuron Shapley: Discovering the Responsible Neurons

Amirata Ghorbani, James Zou|arXiv (Cornell University)|Feb 23, 2020

Adversarial Robustness in Machine Learning52 references61 citations

TL;DR

Neuron Shapley introduces a Shapley-value framework to quantify individual neuron contributions in deep networks, accounting for interactions, and uses a multi-armed bandit approach for efficient estimation. It identifies sparse, critical filters and enables post-training model repair for fairness and robustness without retraining.

ABSTRACT

We develop Neuron Shapley as a new framework to quantify the contribution of individual neurons to the prediction and performance of a deep network. By accounting for interactions across neurons, Neuron Shapley is more effective in identifying important filters compared to common approaches based on activation patterns. Interestingly, removing just 30 filters with the highest Shapley scores effectively destroys the prediction accuracy of Inception-v3 on ImageNet. Visualization of these few critical filters provides insights into how the network functions. Neuron Shapley is a flexible framework and can be applied to identify responsible neurons in many tasks. We illustrate additional applications of identifying filters that are responsible for biased prediction in facial recognition and filters that are vulnerable to adversarial attacks. Removing these filters is a quick way to repair models. Enabling all these applications is a new multi-arm bandit algorithm that we developed to efficiently estimate Neuron Shapley values.

Motivation & Objective

• Quantify each neuron's contribution to a network's performance while accounting for interactions among neurons.
• Develop an efficient algorithm to approximate Neuron Shapley values in large networks.
• Demonstrate that a small set of neurons can dominate network performance across tasks such as accuracy, fairness, and robustness.
• Show that removing a few critical neurons can repair models without retraining.

Proposed method

• Define Neuron Shapley as the unique fair allocation of a network's performance to neurons using V(S) after zeroing out non-S neurons.
• Use Monte-Carlo Shapley estimation with a new Truncated Multi-Armed Bandit (TMAB) approach to identify top-k influential neurons efficiently.
• Introduce early truncation and adaptive sampling to reduce computation by exploiting sparse, high-impact neurons.
• Compare TMAB-Shapley against MC-Shapley showing substantial sample efficiency and high fidelity (R2≈0.975, rank correlation≈0.988).
• Apply the method to Inception-v3 on ImageNet and SqueezeNet on CelebA for tasks on accuracy, fairness, and adversarial vulnerability.
• Demonstrate post-training model repair by zeroing culprit neurons to improve fairness and robustness without retraining.

Experimental results

Research questions

• RQ1 Can Neuron Shapley accurately quantify individual neuron contributions while capturing interactions among neurons?
• RQ2 How can Shapley values be efficiently approximated in large networks to identify a sparse set of critical neurons?
• RQ3 Do a small number of neurons drive overall accuracy, fairness, and robustness, and can removing them repair models without retraining?
• RQ4 Can Neuron Shapley reveal class-specific and layer-specific critical neurons, and how do these relate to interpretability?
• RQ5 Is Neuron Shapley effective for identifying neurons responsible for biases and adversarial vulnerabilities?

Key findings

• A small, sparse set of neurons (filters) largely determines network performance across tasks.
• Removing the top 30 filters with highest Shapley scores substantially destroys Inception-v3 accuracy on ImageNet.
• Zeroing culprits improves fairness in bias-prone tasks and reduces adversarial vulnerability without large accuracy loss.
• Neuron Shapley outperforms alternative neuron-importance methods (e.g., Neuron Conductance) in identifying critical filters for accuracy, fairness, and robustness.
• The TMAB-Shapley algorithm achieves ~10x fewer samples than MC-Shapley while maintaining high accuracy (R2 ≈ 0.975, rank corr ≈ 0.988).
• Class-specific neurons exist, with some filters critical for particular classes while having minimal impact on overall performance when removed.