[Paper Review] Convergence rates for gradient descent in the training of overparameterized artificial neural networks with biases

Arnulf Jentzen, Timo Kröger|arXiv (Cornell University)|Feb 23, 2021

Machine Learning and ELM4 citations

TL;DR

This paper establishes linear convergence rates for batch gradient descent in overparameterized fully-connected ReLU neural networks with biases, showing that training error decays exponentially fast under conditions of sufficient width, small learning rate, and linearly independent input data. The analysis proves that the mean squared error converges to zero at a linear rate, providing theoretical justification for the empirical success of gradient descent in overparameterized settings.

ABSTRACT

In recent years, artificial neural networks have developed into a powerful tool for dealing with a multitude of problems for which classical solution approaches reach their limits. However, it is still unclear why randomly initialized gradient descent optimization algorithms, such as the well-known batch gradient descent, are able to achieve zero training loss in many situations even though the objective function is non-convex and non-smooth. One of the most promising approaches to solving this problem in the field of supervised learning is the analysis of gradient descent optimization in the so-called overparameterized regime. In this article we provide a further contribution to this area of research by considering overparameterized fully-connected rectified artificial neural networks with biases. Specifically, we show that for a fixed number of training data the mean squared error using batch gradient descent optimization applied to such a randomly initialized artificial neural network converges to zero at a linear convergence rate as long as the width of the artificial neural network is large enough, the learning rate is small enough, and the training input data are pairwise linearly independent.

Motivation & Objective

• To explain why gradient descent achieves zero training loss in overparameterized neural networks despite non-convex, non-smooth loss landscapes.
• To extend prior convergence results by incorporating biases in fully-connected ReLU networks.
• To establish quantitative convergence rates for batch gradient descent in this overparameterized regime.
• To analyze the role of random initialization, network width, and input data structure in ensuring convergence.

Proposed method

• Analyzes batch gradient descent in overparameterized fully-connected ReLU networks with biases, using random weight initialization.
• Derives exact expressions for gradients of the empirical risk with respect to network parameters during training.
• Establishes a connection between the training dynamics and the Gram matrix of the network's feature map at initialization.
• Uses concentration inequalities and subexponential tail bounds to control the stochastic Gram matrix during optimization.
• Proves that the empirical risk decays at a linear rate under sufficient width and small learning rate.
• Demonstrates that the limit of the gradient flow corresponds to a linear system governed by the network's feature map and ReLU activation.

Experimental results

Research questions

• RQ1 Under what conditions does batch gradient descent achieve linear convergence in overparameterized ReLU networks with biases?
• RQ2 How does the inclusion of biases affect the convergence behavior of gradient descent in overparameterized settings?
• RQ3 What role does the linear independence of training data play in ensuring convergence to zero training error?
• RQ4 How do the width of the network and the learning rate jointly influence the convergence rate?
• RQ5 Can the dynamics of gradient descent be characterized via a limiting linear system in the overparameterized regime?

Key findings

• The mean squared error converges to zero at a linear rate as the number of training steps increases, provided the network width is sufficiently large.
• Convergence is guaranteed when the learning rate is small enough and the input data are pairwise linearly independent.
• The gradient flow limit corresponds to a linear system where the network's output is determined by the inner product of the input with the network's hidden layer weights.
• The convergence rate is bounded by (1 - ηΛ)^n, where Λ is a positive constant related to the smallest eigenvalue of the deterministic Gram matrix.
• With high probability (at least 1 - ε), the empirical risk decays exponentially fast over time.
• The analysis confirms that the presence of biases does not hinder convergence and that the network's feature map remains well-conditioned during training.