[Paper Review] Natural Neural Networks

Guillaume Desjardins, Karen Simonyan+2 more|arXiv (Cornell University)|Dec 7, 2015

Advanced Neural Network Applications25 references84 citations

TL;DR

This paper introduces Natural Neural Networks, a family of algorithms that accelerate training by adaptively improving the conditioning of the Fisher information matrix through implicit layer-wise whitening of activations. Using the Projected Natural Gradient Descent (PRONG) algorithm, the method efficiently amortizes reparametrization costs and achieves faster convergence in both supervised and unsupervised learning, including large-scale ImageNet training.

ABSTRACT

We introduce Natural Neural Networks, a novel family of algorithms that speed up convergence by adapting their internal representation during training to improve conditioning of the Fisher matrix. In particular, we show a specific example that employs a simple and efficient reparametrization of the neural network weights by implicitly whitening the representation obtained at each layer, while preserving the feed-forward computation of the network. Such networks can be trained efficiently via the proposed Projected Natural Gradient Descent algorithm (PRONG), which amortizes the cost of these reparametrizations over many parameter updates and is closely related to the Mirror Descent online learning algorithm. We highlight the benefits of our method on both unsupervised and supervised learning tasks, and showcase its scalability by training on the large-scale ImageNet Challenge dataset.

Motivation & Objective

• To address slow convergence in neural network training due to poor conditioning of the Fisher information matrix.
• To develop a method that dynamically improves the internal representation of neural networks during training without disrupting feed-forward computation.
• To design an efficient optimization algorithm that amortizes the cost of reparametrization over multiple updates.
• To demonstrate scalability and performance gains on both small-scale and large-scale learning tasks, including ImageNet.
• To preserve the computational efficiency of standard backpropagation while enhancing optimization dynamics.

Proposed method

• The method reparametrizes network weights via implicit whitening of activations at each layer, improving the conditioning of the Fisher matrix.
• It employs the Projected Natural Gradient Descent (PRONG) algorithm, which integrates reparametrization updates into standard optimization steps.
• PRONG is derived as a variant of Mirror Descent, enabling efficient online adaptation of the network's internal representation.
• The reparametrization is applied implicitly, preserving the original feed-forward computation graph and avoiding computational overhead.
• The algorithm amortizes the cost of reparametrization across multiple parameter updates, ensuring efficiency.
• The method maintains full compatibility with standard backpropagation and stochastic gradient descent.

Experimental results

Research questions

• RQ1 Can adaptive reparametrization of neural network weights improve optimization convergence by enhancing Fisher matrix conditioning?
• RQ2 How can reparametrization be applied efficiently during training without disrupting feed-forward computation?
• RQ3 Can the Projected Natural Gradient Descent (PRONG) algorithm effectively amortize reparametrization costs while maintaining training stability?
• RQ4 Does the method scale effectively to large-scale datasets like ImageNet?
• RQ5 How does the implicit whitening of layer representations compare to explicit reparametrization in terms of optimization speed and accuracy?

Key findings

• The proposed Natural Neural Networks achieve faster convergence in both supervised and unsupervised learning tasks by improving the conditioning of the Fisher information matrix.
• The PRONG algorithm enables efficient reparametrization by amortizing computational costs over multiple updates, maintaining high training efficiency.
• The method preserves the feed-forward computation of standard neural networks while enhancing optimization dynamics through implicit whitening.
• The approach scales effectively to large-scale datasets, demonstrating successful training on the ImageNet Challenge dataset.
• The implicit reparametrization improves optimization stability and convergence speed without introducing additional hyperparameters or computational bottlenecks.
• The method shows consistent performance gains across diverse architectures and learning tasks, highlighting its generalizability.