[Paper Review] Scaling Up Exact Neural Network Compression by ReLU Stability

Thiago Serra, Yu, Xin+2 more|arXiv (Cornell University)|Feb 15, 2021

Neural Networks and Applications92 references4 citations

TL;DR

This paper proposes ISA (Identifying Stable Activations), a novel method that accelerates exact neural network compression by solving a single Mixed-Integer Linear Programming (MILP) problem to identify all ReLU neurons with stable (linear) behavior across an input domain. The approach achieves a median 183× speedup over prior state-of-the-art methods on CIFAR-10, enabling exact compression of deeper (5×100) and wider (2×800) networks, with up to 56% of connections removed while preserving accuracy under ℓ1 regularization.

ABSTRACT

We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 56% of the connections on the CIFAR-10 dataset. The code is available at the following link, https://github.com/yuxwind/ExactCompression.

Motivation & Objective

• To address the scalability bottleneck in exact neural network compression, which requires verifying stability of each ReLU neuron across all inputs.
• To reduce the computational cost of identifying stable neurons—those with linear behavior (output always 0 or pre-activation value)—for exact compression.
• To enable practical exact compression of deeper and wider networks (e.g., 5×100, 2×800) on datasets like CIFAR-10 and MNIST.
• To develop a GPU-accelerated preprocessing step that filters out non-stable neurons early, improving overall efficiency.
• To introduce and validate LEO++ as a new compression algorithm that performs layer-wise compression once per layer, rather than per neuron, enhancing scalability.

Proposed method

• Proposes ISA, a method that formulates the stability verification of all ReLU neurons in a feedforward network as a single Mixed-Integer Linear Programming (MILP) problem.
• Leverages a GPU-based preprocessing step to identify neurons that are not stable with respect to the training set, reducing the number of neurons requiring MILP verification.
• Introduces LEO++ as a new compression algorithm that performs lossless compression once per layer by exploiting stable neurons, avoiding per-neuron MILP solving.
• Adapts the MILP formulation to handle input domain constraints, including data and batch normalization layers, by adjusting input bounds accordingly.
• Uses the pre-activation output and ReLU behavior to define linear regions where neuron output is either 0 or linear in input, enabling exact equivalence under domain constraints.
• Employs a hybrid strategy: use preprocessing to eliminate non-stable neurons, then apply MILP to the remaining candidates to verify stability.

Experimental results

Research questions

• RQ1 Can a single MILP formulation be used to verify the stability of all ReLU neurons in a feedforward network, rather than solving one per neuron?
• RQ2 How much faster can exact compression be made by solving one MILP instead of many, especially for deeper and wider networks?
• RQ3 To what extent can exact compression be scaled to deeper (e.g., 5×100) and wider (e.g., 2×800) networks using this approach?
• RQ4 How effective is a GPU-based preprocessing step in reducing the number of neurons requiring MILP verification?
• RQ5 Can LEO++ achieve equivalent compression with significantly less computational cost than per-neuron MILP verification?

Key findings

• The proposed ISA algorithm achieves a median speedup of 183× over the prior state-of-the-art method on the CIFAR-10 dataset, with even greater gains on larger networks.
• On CIFAR-10, the method enables exact compression of 2×800 networks within minutes, achieving up to 56% removal of connections while preserving accuracy under ℓ1 regularization.
• For 2×100 MNIST classifiers trained with ℓ1 regularization that does not degrade accuracy, approximately 20% of neurons and 40% of connections can be removed exactly.
• The GPU preprocessing step alone provides a median 3.2× speedup by filtering out non-stable neurons before MILP solving.
• LEO++ reduces the number of MILP solves from one per stable neuron to one per layer, significantly improving scalability and enabling practical compression of deeper and wider networks.
• On LeNet-based CIFAR-10 models, up to 11.86% of stably inactive neurons can be masked as zero with no accuracy loss, and this pruning is compatible with further lossy sparsification.