[Paper Review] Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee

Alireza Aghasi, Afshin Abdi+2 more|arXiv (Cornell University)|Nov 16, 2016

Stochastic Gradient Optimization Techniques114 citations

TL;DR

Net-Trim introduces a layer-wise convex pruning method that sparsifies trained neural networks while preserving layer input-output relationships within a controlled tolerance, providing performance guarantees and two retraining schemes (parallel and cascade).

ABSTRACT

We introduce and analyze a new technique for model reduction for deep neural networks. While large networks are theoretically capable of learning arbitrarily complex models, overfitting and model redundancy negatively affects the prediction accuracy and model variance. Our Net-Trim algorithm prunes (sparsifies) a trained network layer-wise, removing connections at each layer by solving a convex optimization program. This program seeks a sparse set of weights at each layer that keeps the layer inputs and outputs consistent with the originally trained model. The algorithms and associated analysis are applicable to neural networks operating with the rectified linear unit (ReLU) as the nonlinear activation. We present both parallel and cascade versions of the algorithm. While the latter can achieve slightly simpler models with the same generalization performance, the former can be computed in a distributed manner. In both cases, Net-Trim significantly reduces the number of connections in the network, while also providing enough regularization to slightly reduce the generalization error. We also provide a mathematical analysis of the consistency between the initial network and the retrained model. To analyze the model sample complexity, we derive the general sufficient conditions for the recovery of a sparse transform matrix. For a single layer taking independent Gaussian random vectors of length $N$ as inputs, we show that if the network response can be described using a maximum number of $s$ non-zero weights per node, these weights can be learned from $\\mathcal{O}(s\\log N)$ samples.

Motivation & Objective

• Motivate model reduction to reduce overfitting and redundancy in trained deep networks.
• Develop a layer-wise convex pruning framework that yields sparse weight matrices.
• Provide theoretical guarantees on consistency between the original and retrained networks.
• Offer practical parallel and cascade retraining schemes that are computationally tractable.

Proposed method

• Formulate a convex proxy to prune each layer by minimizing the l1 norm of the weight matrix while enforcing layer-wise consistency through a convex relaxation of the ReLU constraint.
• For a given layer, solve min ||U||1 subject to constraints that approximate consistent post-activation outputs with the original layer output.
• Provide two retraining schemes: parallel Net-Trim (independent layer retraining) and cascade Net-Trim (layer outputs propagated to subsequent retraining).
• Derive theoretical bounds on propagation of retraining error across layers (Theorem 1 and Theorem 2).
• Special case analysis shows sample complexity for learning a sparse weight matrix in a Gaussian input setting (Theorem 3).
• Show practical pruning capability with high sparsity (e.g., >93% links pruned in an example) and compatibility with existing training regularizers.

Experimental results

Research questions

• RQ1 Can layer-wise convex programs recover sparse weight matrices while maintaining similarity between original and retrained layer outputs?
• RQ2 What are the theoretical guarantees on error propagation when retraining layers sequentially or in parallel?
• RQ3 How many samples are required to recover a sparse layer transform under Gaussian input assumptions?
• RQ4 How do parallel and cascade Net-Trim compare in terms of sparsity, feasibility, and generalization performance?
• RQ5 Can Net-Trim be combined post-training with existing regularization techniques without retraining from scratch?

Key findings

• Net-Trim achieves significant sparsification while keeping the network response within a controlled epsilon of the original across layers.
• Parallel Net-Trim retrains each layer independently with a convex program, enabling distributed computation, and yields bounded layerwise error accumulation (sum of epsilons).
• Cascade Net-Trim retrains layers sequentially with inflated tolerances to maintain feasibility, producing potentially sparser models with slightly different error growth.
• For Gaussian inputs, a sparse weight matrix with at most s nonzeros per column can be learned from O(s log N) samples (Theorem 3).
• Net-Trim can post-process already trained networks and further reduce model complexity beyond existing regularizers such as dropout or l1 penalties.
• The framework provides a principled, convex approach to pruning that preserves a close correspondence between the original and retrained networks (consistency guarantees).