[Paper Review] Distilling Critical Paths in Convolutional Neural Networks

Fuxun Yu, Zhuwei Qin+1 more|arXiv (Cornell University)|Oct 28, 2018

Advanced Neural Network Applications5 references20 citations

TL;DR

This paper proposes critical path distillation, a method that identifies and retains only the most class-specific neurons in high-level convolutional layers of CNNs, enabling aggressive model compression without retraining. By analyzing filter activation, gradients, and visualization, it isolates exclusive 'critical paths' for individual classes, achieving 90% fewer neurons and 11.4MB model size with 98.6% average true positive rate on one-vs-all tasks.

ABSTRACT

Neural network compression and acceleration are widely demanded currently due to the resource constraints on most deployment targets. In this paper, through analyzing the filter activation, gradients, and visualizing the filters' functionality in convolutional neural networks, we show that the filters in higher layers learn extremely task-specific features, which are exclusive for only a small subset of the overall tasks, or even a single class. Based on such findings, we reveal the critical paths of information flow for different classes. And by their intrinsic property of exclusiveness, we propose a critical path distillation method, which can effectively customize the convolutional neural networks to small ones with much smaller model size and less computation.

Motivation & Objective

• To investigate the existence of class-specific, exclusive information flow paths in high-level CNN filters.
• To develop a method for compressing CNNs by selectively pruning non-essential filters based on their task-specific contribution.
• To enable efficient, task-specific model deployment by distilling single-class decision pathways without retraining.
• To demonstrate that critical paths are identifiable through activation, gradient, and visualization analysis, enabling structural compression.

Proposed method

• Identifies critical neurons using mean absolute activation across classes to measure filter responsiveness.
• Computes contribution index via first-order Taylor approximation to quantify filter influence on specific class logits.
• Employs activation maximization (AM) visualization to interpret and verify filter functionality and class preference.
• Combines activation, contribution index, and AM results to cross-verify the existence of exclusive critical paths per class.
• Selects filters for distillation based on the product of mean activation and contribution index, with a reserved ratio (e.g., 10%) controlling compression.
• Removes all non-critical filters from high layers, sets logit layer biases to zero, and deploys the distilled network for one-vs-all classification.

Experimental results

Research questions

• RQ1 Do filters in higher CNN layers learn features that are exclusive to a single class or small subset of classes?
• RQ2 Can the information flow for a specific class be traced through a distinct, minimal set of neurons (i.e., a 'critical path')?
• RQ3 Is the exclusiveness of class-specific neurons sufficient to enable effective model distillation without retraining?
• RQ4 Can critical path distillation achieve significant model compression while preserving high classification accuracy on one-vs-all tasks?

Key findings

• The critical path for each class is formed by a small subset of highly specialized, class-exclusive neurons in higher convolutional layers.
• Filters in higher layers exhibit strong class preference, with activation standard deviation increasing toward the top layers, indicating functional specialization.
• The combination of mean activation, contribution index, and AM visualization cross-verified the existence of critical paths for individual classes.
• With a 10% filter retention ratio in the last six convolutional layers, the distilled model achieves an average 98.6% true positive rate across all one-vs-all tasks.
• The distilled model reduces the original 59.0 MB model size to 11.4 MB, achieving a 90% reduction in high-layer neurons.
• The method achieves 11.0% average false positive rate across all one-vs-all tasks without any retraining.