[Paper Review] Variational Dropout Sparsifies Deep Neural Networks

Dmitry Molchanov, Arsenii Ashukha+1 more|arXiv (Cornell University)|Jan 19, 2017

Advanced Neural Network ApplicationsComputer Science31 references348 citations

TL;DR

This paper extends Variational Dropout to allow per-weight dropout rates, introduces Additive Noise Reparameterization and a tight KL-divergence approximation to enable aggressive sparsification, and demonstrates state-of-the-art sparsity with negligible accuracy loss on LeNet and VGG-like networks.

ABSTRACT

We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to 280 times on LeNet architectures and up to 68 times on VGG-like networks with a negligible decrease of accuracy.

Motivation & Objective

• Motivate and leverage Bayesian regularization to induce sparsity in deep neural networks.
• Extend Variational Dropout to unbounded dropout rates to enable Automatic Relevance Determination (ARD) in DNNs.
• Develop low-variance gradient estimators and a tight KL-divergence approximation to train sparse posteriors.
• Demonstrate that Sparse Variational Dropout achieves extreme sparsity in both fully-connected and convolutional layers with minimal accuracy loss.

Proposed method

• Model weights with a variational posterior q(W|θ,α) where α are per-weight dropout rates (p = α/(1+α)).
• Use Additive Noise Reparameterization to replace multiplicative noise with additive noise to reduce gradient variance.
• Apply Local Reparameterization Trick to further reduce gradient variance.
• Provide a tight analytic-like approximation to the KL divergence term over the full α domain to enable training with large dropout rates.
• Derive and employ layer-wise equations for Sparse Variational Dropout in fully-connected and convolutional layers (including convolutional layer formulations).
• Train networks with stochastic variational inference (SGVB) and optimize using Adam, with warm-up strategies for stable sparsification.

Experimental results

Research questions

• RQ1 Can per-weight dropout rates learned via variational inference produce sparse neural networks without substantial accuracy loss?
• RQ2 Does a tight KL-divergence approximation enable stable training when dropout rates are unbounded?
• RQ3 How does Additive Noise Reparameterization impact gradient variance and convergence in Sparse Variational Dropout?
• RQ4 Do fully-connected and convolutional architectures (e.g., LeNet, VGG-like, CIFAR) achieve high sparsity with Sparse Variational Dropout while maintaining performance?

Key findings

• Sparse Variational Dropout yields extremely sparse models in both fully-connected and convolutional layers.
• The method achieves state-of-the-art sparsity on LeNet architectures and scales to larger networks like VGG with negligible accuracy drop.
• Per-weight dropout rates enable ARD-like sparsity without the risks of empirical Bayes.
• Additive Noise Reparameterization and the Local Reparameterization Trick substantially reduce gradient variance and speed up convergence.
• Networks trained with Sparse Variational Dropout resist overfitting on random labels, indicating a favorable generalization behavior.