[Paper Review] Revisiting Natural Gradient for Deep Networks

Razvan Pascanu, Yoshua Bengio|arXiv (Cornell University)|Jan 16, 2013

Stochastic Gradient Optimization TechniquesComputer Science121 citations

TL;DR

This paper revisits natural gradient descent (NGD) for deep networks by linking it to Hessian-Free Optimization and Krylov Subspace Descent, proposing a truncated Newton method to invert the Fisher information matrix instead of using diagonal approximations. The key contribution is a more accurate, second-order-informed NGD variant that improves convergence and generalization, especially when using unlabeled data and robust to training data ordering.

ABSTRACT

We evaluate natural gradient, an algorithm originally proposed in Amari (1997), for learning deep models. The contributions of this paper are as follows. We show the connection between natural gradient and three other recently proposed methods for training deep models: Hessian-Free (Martens, 2010), Krylov Subspace Descent (Vinyals and Povey, 2012) and TONGA (Le Roux et al., 2008). We describe how one can use unlabeled data to improve the generalization error obtained by natural gradient and empirically evaluate the robustness of the algorithm to the ordering of the training set compared to stochastic gradient descent. Finally we extend natural gradient to incorporate second order information alongside the manifold information and provide a benchmark of the new algorithm using a truncated Newton approach for inverting the metric matrix instead of using a diagonal approximation of it.

Motivation & Objective

• To re-evaluate natural gradient descent (NGD) in the context of deep learning, especially its connections to second-order optimization methods.
• To improve NGD by incorporating full second-order information via a truncated Newton method for inverting the Fisher information matrix.
• To investigate the use of unlabeled data to enhance NGD's generalization performance.
• To empirically assess NGD's robustness to training data ordering compared to stochastic gradient descent (SGD).
• To extend NGD with conjugate gradient-like updates using Krylov subspace techniques while reducing memory overhead.

Proposed method

• Establishes theoretical connections between NGD, Hessian-Free Optimization, and Krylov Subspace Descent using the extended Gauss-Newton approximation of the Hessian.
• Proposes a truncated Newton approach to invert the full Fisher information matrix, avoiding diagonal or banded approximations.
• Introduces a manifold-conjugate gradient method that uses linear conjugate gradient to invert the metric, reducing memory usage compared to Krylov subspace methods.
• Employs automatic differentiation to compute efficient Hessian-vector and Jacobian-vector products via forward and backward passes.
• Uses different minibatches for gradient and metric estimation to reduce noise in the natural gradient direction.
• Applies line search and damping to stabilize updates when using small minibatches for gradient and metric estimation.

Experimental results

Research questions

• RQ1 How are natural gradient descent, Hessian-Free Optimization, and Krylov Subspace Descent related through the extended Gauss-Newton approximation?
• RQ2 Can a truncated Newton method for inverting the Fisher matrix improve convergence and generalization in NGD?
• RQ3 Does incorporating unlabeled data into the metric estimation enhance NGD's generalization error?
• RQ4 Is NGD more robust to training data ordering than stochastic gradient descent?
• RQ5 Can a conjugate gradient approach on the parameter manifold outperform standard NGD while reducing memory usage?

Key findings

• The proposed NGD variant using truncated Newton for metric inversion converges faster than standard NGD with diagonal approximations.
• Incorporating unlabeled data into the Fisher information matrix estimation reduces generalization error, especially when the metric is more accurate.
• NGD is empirically more robust to data ordering than SGD, indicating potential advantages in non-stationary learning scenarios.
• The algorithm performs reliably even with small minibatches when gradients and metrics are estimated on separate data, provided damping and learning rate are controlled.
• Using a line search for learning rate and direction in conjugate gradient steps improves performance over fixed or Polak-Ribière-based updates.
• The connection between Krylov Subspace Descent and NGD is validated, with the proposed method achieving similar convergence speed to KSD but with lower memory usage.