[Paper Review] Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition

Lucas Liebenwein, Alaa Maalouf+3 more|arXiv (Cornell University)|Jul 23, 2021

Sparse and Compressive Sensing TechniquesEngineering17 citations

TL;DR

This paper proposes ALDS, a novel global compression framework that automatically determines the optimal per-layer compression ratio for deep neural networks by leveraging layer-wise error bounds derived from the Eckart–Young–Mirsky theorem. By optimizing for minimum maximum error across layers through SVD-based low-rank decomposition with multiple subspaces, ALDS achieves up to 60% parameter compression on ResNet18 (ImageNet), significantly outperforming existing low-rank methods.

ABSTRACT

We present a novel global compression framework for deep neural networks that automatically analyzes each layer to identify the optimal per-layer compression ratio, while simultaneously achieving the desired overall compression. Our algorithm hinges on the idea of compressing each convolutional (or fully-connected) layer by slicing its channels into multiple groups and decomposing each group via low-rank decomposition. At the core of our algorithm is the derivation of layer-wise error bounds from the Eckart Young Mirsky theorem. We then leverage these bounds to frame the compression problem as an optimization problem where we wish to minimize the maximum compression error across layers and propose an efficient algorithm towards a solution. Our experiments indicate that our method outperforms existing low-rank compression approaches across a wide range of networks and data sets. We believe that our results open up new avenues for future research into the global performance-size trade-offs of modern neural networks. Our code is available at https://github.com/lucaslie/torchprune.

Motivation & Objective

• To address the global challenge of balancing compression and accuracy across layers in low-rank neural network compression.
• To develop a framework that jointly optimizes local layer decomposition and global compression ratios for minimal predictive error.
• To derive theoretically grounded error bounds for layer-wise low-rank decomposition using the Eckart–Young–Mirsky theorem.
• To enable automatic, efficient selection of per-layer compression ratios without relying on heuristic or fixed-ratio strategies.
• To explore the impact of multiple subspaces and clustering techniques on compression performance and efficiency.

Proposed method

• The method uses SVD-based low-rank decomposition to compress each convolutional or fully-connected layer by grouping channels into subspaces.
• It introduces a layer-wise decomposition strategy that allows variable numbers of subspaces ($k^\ell$) and ranks ($j^\ell$) per layer to optimize compression and error trade-offs.
• Error bounds are derived using the spectral norm from the Eckart–Young–Mirsky theorem to quantify the maximum relative error per layer.
• The framework formulates the compression problem as an optimization task minimizing the maximum error across all layers, enabling efficient global search.
• An iterative algorithm is used to jointly optimize per-layer decomposition parameters (subspace count and rank) under a global error constraint.
• The method supports both compression-only and one-shot retraining settings, with ablation studies validating component contributions.

Experimental results

Research questions

• RQ1 What is the optimal per-layer compression ratio that minimizes the maximum error across all layers in a neural network?
• RQ2 How does allowing variable numbers of subspaces per layer affect the performance-size trade-off in low-rank compression?
• RQ3 Can error-based allocation of compression ratios outperform fixed-ratio heuristics in maintaining model accuracy?
• RQ4 How do different subspace clustering techniques (e.g., channel slicing vs. projective clustering) impact compression efficiency and accuracy?
• RQ5 What performance gains are achievable by expanding the search space beyond standard SVD decomposition?

Key findings

• ALDS achieves up to 60% parameter compression on ResNet18 (ImageNet), which is three times higher than baseline methods, with minimal accuracy drop.
• The ablation study confirms that variable $k^\ell$ per layer significantly improves performance over fixed $k^\ell$, demonstrating the importance of adaptive decomposition.
• Error-based rank allocation (ALDS-Error) outperforms constant compression ratio heuristics (ALDS-Simple) across all tested settings.
• Using projective clustering (Messi) over channel slicing yields only marginal improvements and incurs high computational cost, making channel slicing preferable for efficiency.
• ALDS+ extends the local search space to include multiple decomposition schemes, resulting in significant performance gains on both CIFAR10 and ImageNet benchmarks.
• The framework enables efficient, automatic, and theoretically grounded compression without requiring specialized hardware or software support.