[Paper Review] Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network

Xiaoxia Wu, Simon S. Du+1 more|arXiv (Cornell University)|Feb 19, 2019

Stochastic Gradient Optimization Techniques33 references44 citations

TL;DR

The paper proves that for two-layer over-parameterized ReLU networks, an adaptive gradient method converges globally in polynomial time, with robustness to hyper-parameter choices and without tuning the learning rate, given sufficient width.

ABSTRACT

Adaptive gradient methods like AdaGrad are widely used in optimizing neural networks. Yet, existing convergence guarantees for adaptive gradient methods require either convexity or smoothness, and, in the smooth setting, only guarantee convergence to a stationary point. We propose an adaptive gradient method and show that for two-layer over-parameterized neural networks -- if the width is sufficiently large (polynomially) -- then the proposed method converges \emph{to the global minimum} in polynomial time, and convergence is robust, \emph{ without the need to fine-tune hyper-parameters such as the step-size schedule and with the level of over-parametrization independent of the training error}. Our analysis indicates in particular that over-parametrization is crucial for the harnessing the full potential of adaptive gradient methods in the setting of neural networks.

Motivation & Objective

• Demonstrate global convergence of an adaptive gradient method in non-convex, over-parameterized neural networks.
• Show that over-parameterization enables robust, hyper-parameter-insensitive convergence.
• Provide a polynomial-time convergence guarantee for an AdaGrad-like adaptive method in this setting.

Proposed method

• Introduce an adaptive gradient method (AdaLoss) as a variant of the norm-based AdaGrad approach.
• Derive a polynomial-time global convergence guarantee under over-parameterization and data-dependent Gram matrix assumptions.
• Prove bounds ensuring the adaptive learning rate remains within a convergent range and does not vanish.
• Use an induction-based proof with carefully constructed hypotheses to bound the evolving learning rate and loss.
• Show that the width m must be sufficiently large to achieve desired convergence guarantees.

Experimental results

Research questions

• RQ1 Can adaptive gradient methods achieve global convergence for non-convex, over-parameterized neural networks?
• RQ2 How does over-parameterization influence the convergence behavior and required learning-rate mechanisms of adaptive methods?
• RQ3 Is there a polynomial-time convergence guarantee for AdaGrad-like methods in this neural network setting, robust to hyper-parameter choices?
• RQ4 What data- and initialization-dependent quantities (e.g., Gram matrices) govern convergence rates?

Key findings

• Gradient descent can achieve improved learning rate with respect to the data-dependent Gram matrix H∞, yielding faster convergence.
• The proposed AdaLoss adaptive method converges to the global minimum in polynomial time under over-parameterization, robust to hyper-parameters.
• Convergence guarantees hold for any choice of hyper-parameters with respect to the rate, though constants vary with those choices.
• Width requirements: m = Ω(n^6 / (λ0^4 δ^3) + η^4 / α^4 · n^4 ||H∞||^4 / (λ0^4 δ^2)).
• The analysis indicates over-parameterization is crucial for harnessing adaptive gradient methods in this neural network setting.