[Paper Review] Spectral-Pruning: Compressing deep neural network via spectral analysis

Taiji Suzuki, Hiroshi Abe+7 more|arXiv (Cornell University)|Aug 26, 2018

Neural Networks and ApplicationsComputer Science38 references15 citations

TL;DR

This paper proposes Spectral-Pruning, a novel model compression method grounded in spectral analysis of internal feature covariance matrices. By defining degrees of freedom via eigenvalue distribution, it establishes a theoretical link between compression and generalization error, achieving superior performance on benchmark datasets through a bias-variance aware pruning strategy.

ABSTRACT

Compression techniques for deep neural network models are becoming very important for the efficient execution of high-performance deep learning systems on edge-computing devices. The concept of model compression is also important for analyzing the generalization error of deep learning, known as the compression-based error bound. However, there is still huge gap between a practically effective compression method and its rigorous background of statistical learning theory. To resolve this issue, we develop a new theoretical framework for model compression and propose a new pruning method called {\it spectral pruning} based on this framework. We define the ``degrees of freedom'' to quantify the intrinsic dimensionality of a model by using the eigenvalue distribution of the covariance matrix across the internal nodes and show that the compression ability is essentially controlled by this quantity. Moreover, we present a sharp generalization error bound of the compressed model and characterize the bias--variance tradeoff induced by the compression procedure. We apply our method to several datasets to justify our theoretical analyses and show the superiority of the the proposed method.

Motivation & Objective

• To bridge the gap between practical model compression techniques and statistical learning theory.
• To develop a theoretically grounded framework for understanding model compression in deep neural networks.
• To characterize how compression affects generalization error through intrinsic model dimensionality.
• To propose a new pruning method that optimizes the bias-variance tradeoff during compression.

Proposed method

• Defining the 'degrees of freedom' of a deep neural network using the eigenvalue distribution of the covariance matrix across internal layers.
• Using spectral analysis of feature activations to quantify the intrinsic dimensionality of the model.
• Deriving a sharp generalization error bound for compressed models based on the degrees of freedom.
• Formulating a pruning strategy that selectively removes parameters based on spectral sensitivity and contribution to model capacity.
• Integrating the theoretical error bound into the pruning objective to balance bias and variance.
• Applying the method to standard deep learning benchmarks to validate theoretical claims.

Experimental results

Research questions

• RQ1 How can model compression be theoretically justified using spectral properties of internal representations?
• RQ2 What is the relationship between the eigenvalue distribution of internal feature covariances and model generalization?
• RQ3 How does compression affect the bias-variance tradeoff in deep neural networks?
• RQ4 Can a pruning method based on spectral analysis outperform existing compression techniques?
• RQ5 To what extent does the proposed theoretical error bound predict generalization performance after compression?

Key findings

• The degrees of freedom, defined via eigenvalue distribution of internal feature covariances, effectively quantify the intrinsic dimensionality of a deep neural network.
• The proposed spectral pruning method achieves state-of-the-art compression performance on multiple benchmark datasets.
• A sharp generalization error bound is derived, linking model compression to generalization error with theoretical rigor.
• The method explicitly models and balances the bias-variance tradeoff induced by compression.
• Empirical results confirm that spectral pruning maintains or improves model accuracy while significantly reducing model size.
• The theoretical framework provides a principled foundation for understanding compression effects on generalization, closing a gap between practice and theory.