[Paper Review] Revisiting the Importance of Individual Units in CNNs via Ablation

Bolei Zhou, Yiyou Sun+2 more|arXiv (Cornell University)|Jun 7, 2018

Adversarial Robustness in Machine LearningComputer Science19 references89 citations

TL;DR

Ablating individual CNN units hardly affects overall accuracy but can cause large drops for specific classes, implying units are specialized for subsets of classes. The study links unit attributes to class-specific importance and examines effects of rotation, batch norm, and dropout.

ABSTRACT

We revisit the importance of the individual units in Convolutional Neural Networks (CNNs) for visual recognition. By conducting unit ablation experiments on CNNs trained on large scale image datasets, we demonstrate that, though ablating any individual unit does not hurt overall classification accuracy, it does lead to significant damage on the accuracy of specific classes. This result shows that an individual unit is specialized to encode information relevant to a subset of classes. We compute the correlation between the accuracy drop under unit ablation and various attributes of an individual unit such as class selectivity and weight L1 norm. We confirm that unit attributes such as class selectivity are a poor predictor for impact on overall accuracy as found previously in recent work \cite{morcos2018importance}. However, our results show that class selectivity along with other attributes are good predictors of the importance of one unit to individual classes. We evaluate the impact of random rotation, batch normalization, and dropout to the importance of units to specific classes. Our results show that units with high selectivity play an important role in network classification power at the individual class level. Understanding and interpreting the behavior of these units is necessary and meaningful.

Motivation & Objective

• Motivate a detailed analysis of how individual CNN units contribute to visual recognition beyond overall accuracy.
• Quantify the impact of ablating single units on both overall and per-class accuracy across large-scale datasets.
• Investigate correlations between unit attributes (selectivity, correlation, L1 norm, concept alignment) and their class-specific importance.
• Examine how training regularizers like batch normalization and dropout affect unit interpretability and class-specific contributions.

Proposed method

• Perform unit ablations by zeroing a unit's weights and biases and measuring validation accuracy drops.
• Compute two types of drops: overall accuracy drop and per-class accuracy drop (max class accuracy drop per unit).
• Rank units by max class accuracy drop to assess class-specific information carried by each unit.
• Analyze correlations between unit attributes (L1 norm, class correlation, class selectivity, concept IoU) and both overall and max class accuracy drops.
• Compare ablations to random rotations of unit directions to distinguish directional phenomena from random directions.
• Evaluate effects of batch normalization and dropout on unit-level interpretability and class-specific contributions.

Experimental results

Research questions

• RQ1 Do single-unit ablations significantly harm overall CNN accuracy or primarily affect a subset of classes?
• RQ2 Which unit attributes best predict a unit’s importance to per-class accuracy and to overall generalization?
• RQ3 How do regularizers like batch normalization and dropout influence the class-specific information carried by units?
• RQ4 Are the effects of ablation attributable to meaningful unit directions rather than random rotations in representation space?

Key findings

• A single unit ablation commonly yields small overall accuracy drops but can cause large drops for certain classes (e.g., >10% for some classes).
• Units tend to specialize, with removed units harming specific classes more than others, indicating alignment with single directions in representation space.
• Max class accuracy drop correlates negatively with several unit attributes, especially class selectivity and class correlation, signaling that more class-aligned units have larger per-class impact.
• L1 norm of a unit correlates with both overall and per-class drops, suggesting pruning-relevant weights relate to unit importance.
• Concept Alignment (IoU with visual concepts) is a strong predictor for which class a unit most affects, outperforming other single-attribute predictors in predicting per-unit class impact.
• Random rotations of unit directions generally show weaker class-specific effects than true unit directions, indicating specialization beyond random directions.