[Paper Review] Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks

Ziwei Ji, Matus Telgarsky|arXiv (Cornell University)|Apr 30, 2020

Stochastic Gradient Optimization TechniquesComputer Science23 references59 citations

TL;DR

This paper demonstrates that gradient descent on two-layer ReLU networks with polylogarithmic width—specifically, width exceeding polylog(n, 1/ε, 1/δ)—can achieve arbitrarily small test error ε within O(1/ε) iterations, provided there are Ω(1/ε²) training examples. The key contribution is showing that such width suffices for generalization, even in the presence of random labels, due to a margin property in the limiting kernel.

ABSTRACT

Recent work has revealed that overparameterized networks trained by gradient descent achieve arbitrarily low training error, and sometimes even low test error. The required width, however, is always polynomial in at least one of the sample size n, the (inverse) training error 1/epsilon, and the (inverse) failure probability 1/delta. This work shows that O(1/epsilon) iterations of gradient descent on two-layer networks of any width exceeding polylog(n, 1/epsilon, 1/delta) and Omega(1/epsilon^2) training examples suffices to achieve a test error of epsilon. The analysis further relies upon a margin property of the limiting kernel, which is guaranteed positive, and can distinguish between true labels and random labels.

Motivation & Objective

• To close the gap between overparameterized networks achieving low training error and achieving low test error in practice.
• To determine whether polylogarithmic width networks can generalize well under gradient descent, rather than requiring polynomial width.
• To analyze the role of the limiting kernel's margin property in distinguishing true labels from random labels.
• To establish that O(1/ε) iterations suffice for test error ε with minimal width requirements.

Proposed method

• Analyzes gradient descent dynamics on two-layer ReLU networks with width in the polylogarithmic regime relative to n, 1/ε, and 1/δ.
• Employs a margin property in the limiting kernel, which is proven to be positive and capable of separating true labels from random labels.
• Uses iterative optimization analysis to show convergence to low test error within O(1/ε) steps.
• Establishes generalization guarantees by linking network width to the kernel's ability to maintain margin under label noise.
• Relies on theoretical analysis of the neural tangent kernel (NTK) regime to derive generalization bounds.

Experimental results

Research questions

• RQ1 Can gradient descent on shallow ReLU networks with polylogarithmic width achieve arbitrarily small test error?
• RQ2 What is the minimal network width required for generalization under gradient descent, beyond polynomial scaling?
• RQ3 How does the margin property of the limiting kernel influence generalization in overparameterized networks?
• RQ4 Can the network generalize even when labels are random, provided the kernel maintains a margin?

Key findings

• Polylogarithmic width networks—specifically, width exceeding polylog(n, 1/ε, 1/δ)—suffice for gradient descent to achieve test error ε.
• O(1/ε) iterations of gradient descent are sufficient to reach test error ε with Ω(1/ε²) training examples.
• The limiting kernel exhibits a positive margin property that enables it to distinguish true labels from random labels.
• Generalization is achieved even under label noise due to the kernel's margin, not just overparameterization.