[Paper Review] DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures

Huanrui Yang, Wei Wen+1 more|arXiv (Cornell University)|Aug 26, 2019

Sparse and Compressive Sensing Techniques53 references41 citations

TL;DR

DeepHoyer introduces differentiable, scale-invariant sparsity regularizers based on the Hoyer measure to achieve element-wise and structural pruning, improving sparsity over prior methods without sacrificing accuracy.

ABSTRACT

In seeking for sparse and efficient neural network models, many previous works investigated on enforcing L1 or L0 regularizers to encourage weight sparsity during training. The L0 regularizer measures the parameter sparsity directly and is invariant to the scaling of parameter values, but it cannot provide useful gradients, and therefore requires complex optimization techniques. The L1 regularizer is almost everywhere differentiable and can be easily optimized with gradient descent. Yet it is not scale-invariant, causing the same shrinking rate to all parameters, which is inefficient in increasing sparsity. Inspired by the Hoyer measure (the ratio between L1 and L2 norms) used in traditional compressed sensing problems, we present DeepHoyer, a set of sparsity-inducing regularizers that are both differentiable almost everywhere and scale-invariant. Our experiments show that enforcing DeepHoyer regularizers can produce even sparser neural network models than previous works, under the same accuracy level. We also show that DeepHoyer can be applied to both element-wise and structural pruning.

Motivation & Objective

• Motivate and develop sparsity-inducing regularizers that are both almost everywhere differentiable and scale-invariant.
• Leverage Hoyer's ratio of l1 to l2 norms to create effective sparsity measures for DNNs.
• Demonstrate that DeepHoyer regularizers yield sparser models than prior methods at similar accuracy.
• Extend the approach to both element-wise and structural (group) pruning.

Proposed method

• Define Hoyer-Square (HS) regularizer H_S(W) = (sum_i |w_i|)^2 / (sum_i w_i^2) for element-wise pruning.
• Propose Group-HS regularizer G_H(W) = ((sum_g ||w^(g)||_2)^2) / (||W||_2^2) for structured pruning, enabling group-level sparsity.
• Integrate DeepHoyer regularizers into standard layer-wise training with objective min L(W) + sum_l (alpha H_S(W^(l)) + beta ||W^(l)||_2) for element-wise pruning.
• Apply Group-HS in a similar layer-wise objective with terms for channel/filter groups and corresponding coefficients alpha_n, alpha_c, alpha_b, beta.
• Train with SGD and perform three-stage pruning: train with regularizer, prune small weights, and finetune with zero elements fixed.
• Demonstrate applicability to both convolutional and fully connected layers, and to ResNet/ AlexNet scales.

Experimental results

Research questions

• RQ1 Can a sparsity regularizer that is both differentiable and scale-invariant outperform l1 or l0-based methods in DNN pruning?
• RQ2 Do Hoyer-inspired regularizers enable more aggressive element-wise sparsity and structural pruning without accuracy loss across standard benchmarks?
• RQ3 How do HS and Group-HS compare to state-of-the-art pruning methods in terms of sparsity, FLOPs reduction, and accuracy?
• RQ4 Is the approach scalable to large networks and datasets (e.g., AlexNet on ImageNet, ResNet on CIFAR-10/ImageNet)?

Key findings

• Hoyer-Square (HS) achieves greater element-wise sparsity than prior methods while preserving accuracy on MNIST LeNet variants and AlexNet.
• HS yields higher compression than original Hoyer and several l1-based baselines across LeNet-300-100, LeNet-5, and AlexNet.
• Group-HS achieves substantial structural pruning gains, reducing FLOPs more than many prior approaches on LeNet-300-100 and LeNet-5, with competitive accuracy.
• On CIFAR-10 and ImageNet, Group-HS-pruned ResNets maintain competitive accuracy while consistently staying above the Pareto frontier for accuracy-FLOPs tradeoffs.