[Paper Review] Ablation Studies in Artificial Neural Networks

Richard Meyes, Melanie Lu+2 more|arXiv (Cornell University)|Jan 24, 2019

Neural Networks and Applications43 references168 citations

TL;DR

This paper conducts single- and pairwise-unit ablations in a shallow MLP on MNIST and in a VGG-19 CNN on ImageNet to study knowledge representations, redundancy, layer/class-specific effects, and recovery training after damage.

ABSTRACT

Ablation studies have been widely used in the field of neuroscience to tackle complex biological systems such as the extensively studied Drosophila central nervous system, the vertebrate brain and more interestingly and most delicately, the human brain. In the past, these kinds of studies were utilized to uncover structure and organization in the brain, i.e. a mapping of features inherent to external stimuli onto different areas of the neocortex. considering the growth in size and complexity of state-of-the-art artificial neural networks (ANNs) and the corresponding growth in complexity of the tasks that are tackled by these networks, the question arises whether ablation studies may be used to investigate these networks for a similar organization of their inner representations. In this paper, we address this question and performed two ablation studies in two fundamentally different ANNs to investigate their inner representations of two well-known benchmark datasets from the computer vision domain. We found that features distinct to the local and global structure of the data are selectively represented in specific parts of the network. Furthermore, some of these representations are redundant, awarding the network a certain robustness to structural damages. We further determined the importance of specific parts of the network for the classification task solely based on the weight structure of single units. Finally, we examined the ability of damaged networks to recover from the consequences of ablations by means of recovery training. We argue that ablations studies are a feasible method to investigate knowledge representations in ANNs and are especially helpful to examine a networks robustness to structural damages, a feature of ANNs that will become increasingly important for future safety-critical applications.

Motivation & Objective

• Motivate the use of neuroscience-inspired ablations to understand knowledge representation in modern ANNs.
• Examine how ablations affect overall and class-specific performance in a shallow MLP and a deep CNN.
• Investigate redundancy and localization of representations across network parts and layers.
• Assess the network’s ability to recover performance via recovery training after damage.

Proposed method

• Perform single-unit ablations in a two-hidden-layer MLP trained on MNIST by zeroing incoming weights to individual units.
• Perform pairwise ablations in the MLP to assess redundancy and combined effects.
• Apply group (1%, 5%, 10%, 25%) ablations to convolutional filters in VGG-19 pre-trained on ImageNet, using similarity-based grouping.
• Evaluate impact using top-1 and top-5 accuracy on ImageNet validation.
• Visualize effects with t-SNE to assess data structure representation post-ablation.
• Conduct recovery training by freezing higher layers and retraining damaged networks to measure restoration of performance.

Experimental results

Research questions

• RQ1 Do ablations reveal localized versus distributed representations in ANNs analogous to neuroscience findings?
• RQ2 How do single-unit and pairwise ablations affect overall and class-specific performance in shallow vs. deep architectures?
• RQ3 Are some layers or classes more vulnerable or more robust to ablations?
• RQ4 Can damaged networks recover performance through targeted recovery training, and to what extent?
• RQ5 Is there evidence of redundancy that buffers against ablations and of selective representations for certain classes?

Key findings

• Single-unit ablations in the MLP can cause large drops in overall accuracy (e.g., up to 44.5 percentage points) but may improve class-specific accuracy for some classes.
• Some units are universally important while others are selectively important for specific classes; the impact correlates with changes in incoming weight distributions after training.
• Pairwise ablations can have stronger effects than the sum of single ablations, indicating redundancies but also potential unique interactions between units.
• In VGG-19, ablation effects depend on layer, with larger portions ablated generally causing bigger accuracy drops; some layers are more critical for certain classes.
• Ablating in some layers can unexpectedly increase performance for some classes, indicating complex redistribution of representations.
• Recovery training can largely restore performance after ablations, sometimes with substantial damage (up to ~80% of filters ablated in a layer) still recovering to near-original levels.