[Paper Review] Analysis of the rate of convergence of an over-parametrized deep neural network estimate learned by gradient descent

Michael Köhler, Adam Krzyżak|arXiv (Cornell University)|Oct 4, 2022

Mathematical Approximation and Integration4 citations

TL;DR

This paper analyzes the rate of convergence of over-parametrized deep neural network estimators trained via gradient descent for nonparametric regression. It establishes that the expected $L_2$ error converges at a rate close to $n^{-1/(1+d)}$ for Hölder-smooth regression functions with exponent $p \in [1/2,1]$, and achieves the optimal $d^*$-dimensional rate in interaction models, under proper initialization and gradient descent hyperparameter choices.

ABSTRACT

Estimation of a regression function from independent and identically distributed random variables is considered. The $L_2$ error with integration with respect to the design measure is used as an error criterion. Over-parametrized deep neural network estimates are defined where all the weights are learned by the gradient descent. It is shown that the expected $L_2$ error of these estimates converges to zero with the rate close to $n^{-1/(1+d)}$ in case that the regression function is Hölder smooth with Hölder exponent $p \in [1/2,1]$. In case of an interaction model where the regression function is assumed to be a sum of Hölder smooth functions where each of the functions depends only on $d^*$ many of $d$ components of the design variable, it is shown that these estimates achieve the corresponding $d^*$-dimensional rate of convergence.

Motivation & Objective

• To theoretically analyze the convergence properties of over-parametrized deep neural networks trained via gradient descent, addressing a gap in existing theory that often ignores over-parametrization and optimization dynamics.
• To establish the rate of convergence of the expected $L_2$ risk for deep neural network estimators in nonparametric regression under realistic training conditions.
• To show that over-parametrized networks trained by gradient descent can achieve near-optimal rates, even when the number of parameters exceeds the sample size.
• To investigate whether such estimators can achieve the $d^*$-dimensional rate in interaction models where the regression function depends on only $d^*$ components of the input.
• To bridge the gap between practical deep learning and theoretical analysis by incorporating realistic training dynamics—random initialization and gradient descent—into convergence rate analysis.

Proposed method

• The authors define an over-parametrized deep neural network estimator where all weights are learned via gradient descent from random initialization.
• They control generalization error using metric entropy bounds on neural network classes with bounded weights, enforced through initialization constraints.
• The optimization process is analyzed by introducing regularization and controlling the change in inner weights during gradient descent via bounded step size and number of steps.
• Approximation error is controlled via over-parametrization and network topology, ensuring that with high probability, a subset of initial inner weights enables good approximation when outer weights are properly trained.
• Theoretical analysis combines metric entropy, Taylor approximation, and piecewise polynomial approximation to bound covering numbers and control generalization error.
• Key technical tools include bounds on derivatives of network functions and covering number estimates for function classes induced by weight constraints.

Experimental results

Research questions

• RQ1 Can over-parametrized deep neural networks trained via gradient descent achieve near-optimal convergence rates in nonparametric regression?
• RQ2 What is the rate of convergence of the expected $L_2$ error for such estimators when the true regression function is Hölder-smooth with exponent $p \in [1/2,1]$?
• RQ3 Does the estimator achieve the $d^*$-dimensional rate of convergence in interaction models where the regression function depends on only $d^*$ input components?
• RQ4 How do initialization constraints and gradient descent hyperparameters (step size, number of steps) affect the generalization and approximation properties?
• RQ5 Can theoretical convergence guarantees be derived for deep networks trained with stochastic or deterministic gradient descent under over-parametrization?

Key findings

• The expected $L_2$ error of the over-parametrized deep neural network estimator trained by gradient descent converges to zero at a rate of order $n^{-1/(1+d) + \epsilon}$ for any $\epsilon > 0$, under Hölder smoothness with exponent $p \in [1/2,1]$.
• In interaction models where the regression function is a sum of $d^*$-variate Hölder-smooth functions, the estimator achieves the $d^*$-dimensional optimal rate of convergence $n^{-2p/(d^*+1)}$.
• The convergence rate is nearly optimal, approaching $n^{-1/(1+d)}$, which matches the minimax rate for nonparametric regression under Hölder smoothness.
• The analysis shows that with high probability, a subset of randomly initialized inner weights enables good approximation, provided the outer weights are trained appropriately.
• The number of gradient descent steps and step size are carefully chosen to prevent large changes in inner weights, ensuring stability and generalization.
• Theoretical bounds rely on metric entropy and covering number estimates for neural network function classes with bounded weights, combined with Taylor-based approximation arguments.