[Paper Review] On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective

Lili Su, Pengkun Yang|arXiv (Cornell University)|Jan 1, 2019

Stochastic Gradient Optimization Techniques22 citations

TL;DR

This paper studies gradient descent training of over-parameterized two-layer ReLU networks by framing the optimization process as applying powers of an integral operator derived from the input distribution. It shows that with width Ω(n log n), the empirical risk converges linearly to the low-rank approximation error of the target function, achieving a rate independent of sample size n, and proves nearly-linear over-parameterization suffices for fast convergence.

ABSTRACT

We consider training over-parameterized two-layer neural networks with Rectified Linear Unit (ReLU) using gradient descent (GD) method. Inspired by a recent line of work, we study the evolutions of network prediction errors across GD iterations, which can be neatly described in a matrix form. When the network is sufficiently over-parameterized, these matrices individually approximate {\em an} integral operator which is determined by the feature vector distribution $ ho$ only. Consequently, GD method can be viewed as {\em approximately} applying the powers of this integral operator on the underlying/target function $f^*$ that generates the responses/labels. We show that if $f^*$ admits a low-rank approximation with respect to the eigenspaces of this integral operator, then the empirical risk decreases to this low rank approximation error at a linear rate which is determined by $f^*$ and $ ho$ only, i.e., the rate is independent of the sample size $n$. Furthermore, if $f^*$ has zero low-rank approximation error, then, as long as the width of the neural network is $\Omega(n\log n)$, the empirical risk decreases to $\Theta(1/\sqrt{n})$. To the best of our knowledge, this is the first result showing the sufficiency of nearly-linear network over-parameterization. We provide an application of our general results to the setting where $ ho$ is the uniform distribution on the spheres and $f^*$ is a polynomial. Throughout this paper, we consider the scenario where the input dimension $d$ is fixed.

Motivation & Objective

• To understand the generalization and optimization dynamics of over-parameterized two-layer ReLU networks under gradient descent.
• To characterize how the evolution of prediction error relates to an integral operator induced by the input distribution.
• To establish conditions under which gradient descent achieves fast, linear convergence to the low-rank approximation of the target function.
• To determine the minimal over-parameterization level required for convergence, particularly in terms of network width relative to sample size.

Proposed method

• Modeling the prediction error evolution across GD iterations using matrix representations that reflect the integral operator structure.
• Showing that, in the over-parameterized regime, the Hessian-like matrices converge to an integral operator determined solely by the input distribution $\rho$.
• Analyzing the GD dynamics as approximately applying powers of this integral operator to the target function $f^*$.
• Establishing that convergence speed depends only on $f^*$ and $\rho$, not on the sample size $n$, via spectral analysis of the integral operator.
• Proving that when $f^*$ admits a low-rank approximation in the eigenspaces of the integral operator, the empirical risk decays linearly.
• Demonstrating that with network width $\Omega(n\log n)$, the risk converges to $\Theta(1/\sqrt{n})$ when the low-rank error is zero.

Experimental results

Research questions

• RQ1 How does gradient descent optimize over-parameterized ReLU networks in terms of functional approximation?
• RQ2 What role does the input distribution $\rho$ play in determining the convergence rate of gradient descent?
• RQ3 Can nearly-linear over-parameterization ($\Omega(n\log n)$) suffice for fast convergence to the target function?
• RQ4 Under what conditions does the empirical risk decay linearly, independent of sample size $n$?
• RQ5 How does the spectral structure of the target function $f^*$ relative to the integral operator affect optimization dynamics?

Key findings

• When the target function $f^*$ admits a low-rank approximation in the eigenspaces of the integral operator, the empirical risk decreases linearly with a rate determined only by $f^*$ and $\rho$, not by the sample size $n$.
• With network width $\Omega(n\log n)$, the empirical risk converges to $\Theta(1/\sqrt{n})$ when the low-rank approximation error of $f^*$ is zero.
• The optimization dynamics of gradient descent are approximately equivalent to applying powers of an integral operator derived from the input distribution $\rho$.
• The convergence rate is independent of the number of samples $n$, highlighting a fundamental property of over-parameterized networks.
• The analysis reveals that nearly-linear over-parameterization is sufficient for fast convergence, which is a new result in the literature.
• The results are established under fixed input dimension $d$, focusing on the interplay between network width, sample size, and function complexity.