[Paper Review] On the Convergence of Deep Networks with Sample Quadratic Overparameterization.

Asaf Noy, Yi Xu+2 more|arXiv (Cornell University)|Jan 12, 2021

Stochastic Gradient Optimization Techniques1 citations

TL;DR

This paper establishes convergence to a global minimum for ReLU networks with width quadratic in sample size and linear in depth, achieving logarithmic convergence time. It introduces a surrogate network with fixed activation patterns that enables tighter theoretical bounds and finite-width equivalence to the Neural Tangent Kernel, offering a novel training acceleration method with implications for generalization.

ABSTRACT

Deep neural networks' remarkable ability to correctly fit training data when optimized by gradient-based algorithms is yet to be fully understood. Recent theoretical results explain the convergence for ReLU networks that are wider than those used in practice by orders of magnitude. In this work, we take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time. We show that convergence to a global minimum is guaranteed for networks with widths quadratic in the sample size and linear in their depth at a time logarithmic in both. Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size. This construction can be viewed as a novel technique to accelerate training, while its tight finite-width equivalence to Neural Tangent Kernel (NTK) suggests it can be utilized to study generalization as well.

Motivation & Objective

• To close the gap between theoretical convergence guarantees and practical deep learning by improving theoretical bounds on network width and convergence time.
• To provide a rigorous theoretical analysis of ReLU networks under realistic width and depth scaling, closer to practical settings.
• To develop a surrogate network construction that maintains finite-width equivalence to the Neural Tangent Kernel (NTK) while enabling tighter convergence analysis.
• To demonstrate that training can be accelerated via the surrogate network without sacrificing theoretical equivalence to the NTK.
• To lay theoretical groundwork for studying generalization in finite-width ReLU networks using the proposed construction.

Proposed method

• The authors construct a surrogate network with fixed activation patterns that can be transformed into an equivalent ReLU network at any time.
• This surrogate network is designed to maintain finite-width equivalence to the Neural Tangent Kernel (NTK), enabling theoretical analysis.
• The method leverages the structure of ReLU networks and activation pattern stability to derive convergence bounds under realistic width scaling.
• Convergence is proven via analysis of the network's dynamics under gradient descent, showing logarithmic convergence time in both width and depth.
• The construction allows for tighter theoretical bounds than prior work by avoiding overparameterization assumptions that are orders of magnitude larger than practical settings.
• The surrogate network serves as a tool for both accelerating training and enabling generalization analysis through NTK equivalence.

Experimental results

Research questions

• RQ1 Can global convergence be guaranteed for ReLU networks with width scaling closer to practical settings, rather than requiring extreme overparameterization?
• RQ2 What is the minimal network width and depth required to ensure convergence to a global minimum under gradient descent?
• RQ3 How can a surrogate network with fixed activation patterns be constructed to maintain finite-width equivalence to the NTK?
• RQ4 Can such a surrogate network be used to accelerate training while preserving theoretical guarantees?
• RQ5 What are the implications of this construction for understanding generalization in finite-width deep networks?

Key findings

• Convergence to a global minimum is guaranteed for ReLU networks with width quadratic in the sample size and linear in depth.
• The convergence time scales logarithmically in both the network width and depth, significantly improving prior bounds.
• The proposed surrogate network maintains finite-width equivalence to the Neural Tangent Kernel (NTK), enabling theoretical analysis of generalization.
• The construction allows for a tighter theoretical analysis by avoiding the extreme overparameterization assumed in prior works.
• The surrogate network can be transformed into an equivalent ReLU network at any time, providing a practical mechanism for training acceleration.
• The method establishes a new theoretical framework for studying generalization in finite-width deep networks through NTK equivalence.