[Paper Review] Deep Net Triage: Analyzing the Importance of Network Layers via Structural Compression

Theodore S. Nowak, Jason J. Corso|arXiv (Cornell University)|Jan 15, 2018

COVID-19 diagnosis using AI11 references3 citations

TL;DR

This paper introduces 'deep net triage,' a method to assess the criticality of individual layers in deep neural networks by applying structural compression—replacing blocks of layers with a single compressed layer. The key finding is that no single layer is more critical than others, and fine-tuning the entire compressed network can yield performance equal to or better than the original, while Knowledge Distillation accelerates convergence without exceeding the teacher model's accuracy.

ABSTRACT

Despite their prevalence, deep networks are poorly understood. This is due, at least in part, to their highly parameterized nature. As such, while certain structures have been found to work better than others, the significance of a model's unique structure, or the importance of a given layer, and how these translate to overall accuracy, remains unclear. In this paper, we analyze these properties of deep neural networks via a process we term deep net triage. Like medical triage---the assessment of the importance of various wounds---we assess the importance of layers in a neural network, or as we call it, their criticality. We do this by applying structural compression, whereby we reduce a block of layers to a single layer. After compressing a set of layers, we apply a combination of initialization and training schemes, and look at network accuracy, convergence, and the layer's learned filters to assess the criticality of the layer. We apply this analysis across four data sets of varying complexity. We find that the accuracy of the model does not depend on which layer was compressed; that accuracy can be recovered or exceeded after compression by fine-tuning across the entire model; and, lastly, that Knowledge Distillation can be used to hasten convergence of a compressed network, but constrains the accuracy attainable to that of the base model.

Motivation & Objective

• To understand the relative importance of individual layers in deep neural networks, which remain poorly understood despite widespread use.
• To develop a systematic method for assessing layer criticality without relying on existing pruning or compression techniques.
• To evaluate how structural compression affects model accuracy, convergence speed, and learned filter representations.
• To investigate whether Knowledge Distillation can accelerate training of compressed networks and whether it enables performance beyond the teacher model.
• To provide a structured, interpretable framework for analyzing deep network components using compression and distillation.

Proposed method

• Structural compression is applied by replacing a block of consecutive layers (e.g., two to three convolutions followed by a pooling layer) with a single equivalent layer.
• The compressed network is initialized using random weights, average weights from the parent network, or via a Student-Teacher network (STN) with intermediate checkpoints.
• Training is performed either only on the compressed layer or across the entire network after initialization to assess convergence and accuracy.
• Knowledge Distillation is used to transfer knowledge from the parent model to the compressed student model to accelerate convergence.
• Filter activation visualizations are used to qualitatively compare learned features across models and validate structural changes.
• The analysis is applied across four datasets of increasing complexity: MNIST, CIFAR10, CIFAR100, and Stanford Dogs.

Experimental results

Research questions

• RQ1 Does the choice of which layer is compressed affect the final model accuracy?
• RQ2 Can fine-tuning the entire compressed network recover or exceed the accuracy of the original model?
• RQ3 Does Knowledge Distillation improve convergence speed in compressed networks, and does it allow the student to surpass the teacher’s performance?
• RQ4 How do filter representations in compressed networks compare to those in the original and distilled models?
• RQ5 Does the complexity of the dataset influence the criticality or behavior of compressed layers?

Key findings

• No layer is more critical than another: compressing any layer results in similar maximal attainable accuracy across all datasets.
• Fine-tuning the entire compressed network leads to novel, more optimal filter representations, resulting in performance that exceeds the baseline model.
• Knowledge Distillation significantly reduces the number of epochs required for convergence, but the student network cannot achieve higher accuracy than the teacher model.
• Filter visualizations show that STN networks learn responses nearly identical to the parent model, while TM-RW networks learn distinct, more optimal filters, explaining performance gains.
• Convergence speed varies by data complexity: STN models converge faster on complex datasets like Stanford Dogs, suggesting distillation aids training stability in harder regimes.
• Random or average-weight initialization does not improve convergence and may even slow it down, indicating that proper initialization is crucial for compressed models.