[Paper Review] Model Preserving Compression for Neural Networks

Jerry Chee, Megan Renz+2 more|arXiv (Cornell University)|Jul 30, 2021

Seismic Imaging and Inversion Techniques4 citations

TL;DR

This paper introduces a model-preserving compression method for neural networks using interpolative decomposition (ID) to simultaneously prune and correct channel activations, preserving the original model's per-example decisions and structure without fine-tuning. The approach achieves high prediction correlation (97.3%) with the original model on CIFAR-10 and competitive accuracy on ImageNet, while enabling automatic per-layer compression sizing and theoretical analysis.

ABSTRACT

After training complex deep learning models, a common task is to compress the model to reduce compute and storage demands. When compressing, it is desirable to preserve the original model's per-example decisions (e.g., to go beyond top-1 accuracy or preserve robustness), maintain the network's structure, automatically determine per-layer compression levels, and eliminate the need for fine tuning. No existing compression methods simultaneously satisfy these criteria $\unicode{x2014}$ we introduce a principled approach that does by leveraging interpolative decompositions. Our approach simultaneously selects and eliminates channels (analogously, neurons), then constructs an interpolation matrix that propagates a correction into the next layer, preserving the network's structure. Consequently, our method achieves good performance even without fine tuning and admits theoretical analysis. Our theoretical generalization bound for a one layer network lends itself naturally to a heuristic that allows our method to automatically choose per-layer sizes for deep networks. We demonstrate the efficacy of our approach with strong empirical performance on a variety of tasks, models, and datasets $\unicode{x2014}$ from simple one-hidden-layer networks to deep networks on ImageNet.

Motivation & Objective

• To develop a compression method that preserves per-example decisions, network structure, and enables automatic per-layer compression sizing.
• To eliminate the need for fine-tuning while maintaining high model fidelity after compression.
• To provide a theoretically grounded method for channel pruning with interpretable correction mechanisms via interpolative decomposition.
• To enable practical deployment of compressed models by ensuring compatibility with existing ML pipelines and fairness across classes.
• To introduce a new metric for measuring model similarity based on prediction correlation to evaluate preservation of original behavior.

Proposed method

• Uses interpolative decomposition (ID) to represent pruned channels as linear combinations of retained channels, enabling correction propagation across layers.
• Applies ID to post-activation outputs to simultaneously select and eliminate channels while updating remaining weights to correct for removed neurons.
• Constructs an interpolation matrix that fuses information from pruned channels into the next layer, preserving network structure with reduced layer sizes.
• Employs unlabeled data to compute ID approximations, avoiding reliance on labeled data or fine-tuning.
• Derives a theoretical generalization bound for one-layer networks to inform a heuristic for automatic per-layer compression size selection.
• Introduces a prediction correlation metric to quantitatively assess preservation of original model decisions post-compression.

Experimental results

Research questions

• RQ1 Can a compression method preserve per-example decisions without fine-tuning, ensuring robustness and fairness?
• RQ2 How can interpolative decomposition be used to simultaneously prune and correct for channel removal in deep networks?
• RQ3 Can theoretical bounds on generalization inform automatic per-layer compression sizing in deep architectures?
• RQ4 How does the method perform in terms of prediction correlation and accuracy when compared to fine-tuned baselines?
• RQ5 Is the method robust to missing data classes in the pruning set, particularly for fairness and generalization?

Key findings

• The proposed ID-based compression method achieves 97.3% prediction correlation with the original model on CIFAR-10 after pruning to 50% FLOPs, significantly outperforming fine-tuned baselines (93.2% correlation).
• On ImageNet, the method maintains pre-fine-tuning accuracy of 68.06% with a 10k pruning set, showing minimal sensitivity to pruning set size.
• The method preserves per-class accuracy even when one class is excluded from the pruning set, maintaining reasonable performance on unrepresented classes (e.g., 85.2% accuracy on missing class in CIFAR-10).
• Theoretical analysis for one-layer networks supports the use of ID-based correction and enables a heuristic for automatic per-layer compression sizing.
• Compression costs (including ID computation) are negligible compared to fine-tuning, with ID computation requiring only 1,000 data points and minimal runtime.
• The method is compatible with other compression techniques due to its structural preservation, enabling composability in practical pipelines.