[Paper Review] Geometry of Optimization and Implicit Regularization in Deep Learning

Behnam Neyshabur, Ryota Tomioka+2 more|arXiv (Cornell University)|May 8, 2017

Advanced Numerical Analysis Techniques5 references89 citations

TL;DR

The paper argues that optimization geometry induces implicit regularization in deep networks, showing that generalization is governed by optimization dynamics rather than network size, and introduces Path-SGD as a rescaling-invariant, path-regularized optimization method.

ABSTRACT

We argue that the optimization plays a crucial role in generalization of deep learning models through implicit regularization. We do this by demonstrating that generalization ability is not controlled by network size but rather by some other implicit control. We then demonstrate how changing the empirical optimization procedure can improve generalization, even if actual optimization quality is not affected. We do so by studying the geometry of the parameter space of deep networks, and devising an optimization algorithm attuned to this geometry.

Motivation & Objective

• Motivate the role of optimization as an implicit regularizer in deep learning.
• Investigate how network size relates to generalization under different optimization dynamics.
• Propose a geometry-aware optimization method to improve generalization in RELU networks.

Proposed method

• Introduce path-based regularization via the path-norm and path-regularizer defined from products of weights along input-to-output paths.
• Define rescaling invariance and show that standard SGD/gradient descent are not rescaling invariant for RELU networks.
• Derive the Path-SGD update as an approximate steepest descent with respect to the path-regularizer.
• Provide an efficient forward-backward algorithm to compute Path-SGD updates without enumerating all paths.
• Demonstrate, via experiments on MNIST, CIFAR-10/100, and SVHN, that Path-SGD can converge faster and yield better generalization compared to SGD and AdaGrad (with and without dropout).

Experimental results

Research questions

• RQ1 Does network size alone control generalization in deep networks, or is there an implicit capacity control driven by optimization?
• RQ2 Can a geometry-aware optimization method align with inductive biases to improve generalization in RELU networks?
• RQ3 Is it possible to construct a rescaling-invariant optimization algorithm that avoids issues from unbalanced networks?
• RQ4 Does path-based regularization lead to better implicit regularization and generalization than traditional weight decay?
• RQ5 How does Path-SGD compare to standard optimizers on common benchmarks?

Key findings

• Increasing network size can continue to reduce training error while test error may still decrease, indicating implicit regularization from optimization.
• Path-SGD, a path-regularized, rescaling-invariant update, often reaches similar or better objective and generalizes better than SGD or AdaGrad.
• Path-SGD updates can be implemented efficiently in a forward-backward pass with mini-batches, making it practical for standard training cycles.
• Experiments show Path-SGD outperforms baselines on MNIST, CIFAR-10/100, and SVHN in terms of training speed and generalization under various settings (balanced/unbalanced initializations, with/without dropout).
• The path-regularizer is invariant to rescaling, and Path-SGD is proven to be rescaling invariant, reinforcing its suitability for RELU networks.