[Paper Review] Data-Independent Structured Pruning of Neural Networks via Coresets

Ben Mussay, Daniel E. Feldman+3 more|arXiv (Cornell University)|Aug 19, 2020

Adversarial Robustness in Machine Learning45 references4 citations

TL;DR

This paper proposes a data-independent structured pruning method for neural networks using the coreset framework, which provably guarantees approximation error bounds for any input, including adversarial examples. By constructing coresets of neurons/filters layer-by-layer, the method achieves high compression rates with strong theoretical guarantees and faster training than prior approaches.

ABSTRACT

Model compression is crucial for deployment of neural networks on devices with limited computational and memory resources. Many different methods show comparable accuracy of the compressed model and similar compression rates. However, the majority of the compression methods are based on heuristics and offer no worst-case guarantees on the trade-off between the compression rate and the approximation error for an arbitrarily new sample. We propose the first efficient structured pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Our method is based on the coreset framework and it approximates the output of a layer of neurons/filters by a coreset of neurons/filters in the previous layer and discards the rest. We apply this framework in a layer-by-layer fashion from the bottom to the top. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input $x\in \mathbb{R}^d$, including an adversarial one.

Motivation & Objective

• To address the lack of worst-case guarantees in existing model compression methods, especially regarding approximation error for unseen or adversarial inputs.
• To develop a structured pruning framework that ensures provable trade-offs between compression rate and approximation error.
• To enable efficient, one-shot compression with minimal fine-tuning by improving the initial approximation of the compressed model.
• To ensure data independence so that the method performs reliably across all inputs, not just training or validation data.
• To combine theoretical guarantees with practical efficiency for real-world deployment on resource-constrained devices.

Proposed method

• The method applies the coreset framework to neural network layers, treating neurons or filters as points to be approximated by a smaller subset.
• For each layer, it constructs a coreset of neurons/filters from the previous layer that provably approximates the output of the current layer.
• The coreset algorithm selects a subset of neurons/filters and assigns new weights to them to maintain approximation quality.
• The framework is applied layer-by-layer from the bottom to the top of the network, enabling end-to-end compression.
• The approach supports both fully connected and convolutional layers, with specific coreset constructions for common activation functions like ReLU.
• The method is one-shot and data-independent, meaning it does not rely on training data distribution for guarantees.

Experimental results

Research questions

• RQ1 Can a structured pruning method provide provable worst-case guarantees on approximation error for any input, including adversarial examples?
• RQ2 How can the coreset framework be adapted to neural network pruning to ensure both high compression and strong theoretical bounds?
• RQ3 Does a data-independent pruning method outperform data-dependent methods in terms of robustness and generalization to unseen inputs?
• RQ4 To what extent does a high-quality initial approximation reduce fine-tuning time in compressed models?
• RQ5 Can the coreset-based approach achieve comparable accuracy to state-of-the-art methods while offering stronger theoretical guarantees?

Key findings

• The proposed method achieved 70% compression on VGG-19 (CIFAR-10) with a small model accuracy of 6.02%, outperforming prior methods in approximation quality before fine-tuning.
• For ResNet-56, the method achieved 40% compression with a top-1 accuracy of 25.11%, matching or exceeding state-of-the-art results while ensuring theoretical guarantees.
• On ResNet-50 (ImageNet), the method achieved 62% parameter reduction with a top-1 accuracy of 25.11%, demonstrating scalability to large models.
• The initial approximation via coresets reduced fine-tuning time significantly, as shown in Figure 7, where accuracy converged faster than with random initialization.
• The method outperformed [36] by 12% in accuracy improvement before fine-tuning on VGG-19, indicating a superior initial model approximation.
• The framework is one-shot and computationally efficient, with fast coreset construction and minimal need for iterative refinement.