[Paper Review] Learning the Number of Neurons in Deep Networks

José M. Alvarez, Mathieu Salzmann|arXiv (Cornell University)|Nov 19, 2016

Machine Learning and Algorithms197 citations

TL;DR

The paper introduces a group sparsity regularizer to automatically determine and prune the number of neurons per layer during training, starting from an overcomplete network, achieving substantial parameter reductions with maintained or improved accuracy.

ABSTRACT

Nowadays, the number of layers and of neurons in each layer of a deep network are typically set manually. While very deep and wide networks have proven effective in general, they come at a high memory and computation cost, thus making them impractical for constrained platforms. These networks, however, are known to have many redundant parameters, and could thus, in principle, be replaced by more compact architectures. In this paper, we introduce an approach to automatically determining the number of neurons in each layer of a deep network during learning. To this end, we propose to make use of structured sparsity during learning. More precisely, we use a group sparsity regularizer on the parameters of the network, where each group is defined to act on a single neuron. Starting from an overcomplete network, we show that our approach can reduce the number of parameters by up to 80\% while retaining or even improving the network accuracy.

Motivation & Objective

• Motivate automatic determination of layer widths to reduce memory and compute costs.
• Propose a regularized learning framework that can remove entire neurons during training.
• Enable end-to-end learning of both network parameters and neuron counts without pre-training the network.
• Demonstrate scalability to large datasets and architectures on ImageNet and ICDAR.

Proposed method

• Define neurons in a layer as groups of parameters and apply a group sparsity regularizer to drive some groups to zero.
• Use a sparse group Lasso variant that combines group-norm and L1 penalties to encourage both group-level and within-group sparsity.
• Optimize with proximal gradient descent; apply the proximal operator per neuron group to zero out entire neurons.
• Start from an overcomplete network and iteratively prune neurons during learning to obtain a compact architecture.
• Experiment with two regularizer variants: Eq. (2) group sparsity and Eq. (3) sparse group Lasso with alpha parameter.

Experimental results

Research questions

• RQ1 Can group sparsity regularization automatically reduce the number of neurons per layer during learning?
• RQ2 How much parameter reduction is achievable without sacrificing accuracy on large-scale datasets?
• RQ3 Does the method generalize across different architectures and datasets (ImageNet, Places2, ICDAR)?
• RQ4 What are the test-time benefits (speed and memory) of the pruned networks?
• RQ5 How sensitive is the method to regularization hyperparameters across layers?

Key findings

• Starting from overcomplete networks, the method can reduce parameters by up to 80%.
• Most methods maintain or improve top-1 accuracy after pruning, with gains up to 2.45% on ImageNet in some settings.
• BNet C with GS achieves 12.7% neuron reduction and 27.4% total parameter reduction, while preserving accuracy within 1.6% of the baseline.
• Dec 8 with larger initial widths (M=640/768) yields 19–26% neuron reduction and up to 48% parameter reduction, with accuracy comparable or better than baseline.
• On ICDAR, Dec 3 with GS/SGL reduces parameters by up to 80% and can improve accuracy over the MaxPool/MaxOut baselines.
• At test time, pruned networks show significant speedups (up to ~50% in some cases) and memory savings (up to 82% in parameters).