[Paper Review] Dynamical Isometry: The Missing Ingredient for Neural Network Pruning

Huan Wang, Can Qin+2 more|arXiv (Cornell University)|May 12, 2021

Neural Networks and Applications55 references4 citations

TL;DR

This paper identifies dynamical isometry degradation after pruning as the root cause of poor finetuning performance and proposes OrthP, a plug-and-play method to recover it. It explains why larger finetuning learning rates improve accuracy and resolves the debate on pruning's value by showing that suboptimal finetuning settings—especially low learning rates—lead to misleading comparisons with scratch training.

ABSTRACT

Several recent works [40, 24] observed an interesting phenomenon in neural network pruning: A larger finetuning learning rate can improve the final performance significantly. Unfortunately, the reason behind it remains elusive up to date. This paper is meant to explain it through the lens of dynamical isometry [42]. Specifically, we examine neural network pruning from an unusual perspective: pruning as initialization for finetuning, and ask whether the inherited weights serve as a good initialization for the finetuning? The insights from dynamical isometry suggest a negative answer. Despite its critical role, this issue has not been well-recognized by the community so far. In this paper, we will show the understanding of this problem is very important -- on top of explaining the aforementioned mystery about the larger finetuning rate, it also unveils the mystery about the value of pruning [5, 30]. Besides a clearer theoretical understanding of pruning, resolving the problem can also bring us considerable performance benefits in practice.

Motivation & Objective

• To explain the empirical observation that larger finetuning learning rates significantly improve pruned model performance.
• To resolve the controversy over whether pruning provides real value compared to training small models from scratch.
• To investigate whether pruning serves as a good initialization for finetuning by analyzing signal propagation via dynamical isometry.
• To propose a method for recovering dynamical isometry after pruning to improve finetuning convergence and final accuracy.

Proposed method

• Analyzes dynamical isometry using mean Jacobian singular values (JSV) to measure signal propagation in pruned networks.
• Proposes OrthP, a method that orthogonalizes the pruned network's weights to restore dynamical isometry before finetuning.
• Evaluates the impact of different finetuning learning rates on dynamical isometry recovery and final accuracy.
• Compares pruned models finetuned with optimal vs. suboptimal learning rates against models trained from scratch.
• Tests OrthP on non-linear and convolutional architectures (e.g., LeNet-5-ReLU) to assess generalization beyond linear models.
• Uses signal propagation analysis to diagnose how pruning disrupts gradient flow and why standard finetuning fails to recover it.

Experimental results

Research questions

• RQ1 Why does a larger finetuning learning rate lead to significantly better performance in pruned neural networks?
• RQ2 Is pruning truly beneficial compared to training small models from scratch, or is the comparison biased by suboptimal finetuning settings?
• RQ3 To what extent does pruning degrade dynamical isometry, and can this degradation be reversed before finetuning?
• RQ4 Can a plug-and-play method like OrthP effectively restore dynamical isometry and improve finetuning performance in practical networks?
• RQ5 How does the interplay between network architecture, pruning ratio, and finetuning hyperparameters affect dynamical isometry recovery?

Key findings

• Pruning significantly degrades dynamical isometry, as measured by a drop in mean Jacobian singular values (JSV), especially at higher pruning ratios.
• A larger finetuning learning rate accelerates the recovery of dynamical isometry, explaining its performance boost in pruned models.
• The performance gap between pruning and scratch training is largely due to suboptimal finetuning learning rates in prior works, not inherent pruning limitations.
• With optimal finetuning learning rates, even basic L1-norm pruning outperforms training from scratch across multiple pruning ratios.
• OrthP successfully recovers dynamical isometry in linear and simple non-linear networks, but its effectiveness diminishes in deeper or more complex architectures like LeNet-5-ReLU.
• The study reveals that finetuning is not a post-processing step but a critical recovery phase, and dynamical isometry recovery should be prioritized in pruning pipelines.