[Paper Review] Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions

Stefano Sarao Mannelli, Eric Vanden‐Eijnden+1 more|arXiv (Cornell University)|Jun 27, 2020

Stochastic Gradient Optimization Techniques30 references16 citations

TL;DR

This paper studies one-hidden-layer neural networks with quadratic activation functions in the over-parametrized regime, showing that gradient descent on the empirical loss converges to the teacher network with high probability when the number of training samples exceeds a critical threshold $\alpha_c = m^* + 1$ per input dimension. It establishes that generalization error vanishes under this condition and derives explicit convergence rates, including $O(t^{-2})$ and exponential decay depending on model dimensions.

ABSTRACT

We study the dynamics of optimization and the generalization properties of one-hidden layer neural networks with quadratic activation function in the over-parametrized regime where the layer width $m$ is larger than the input dimension $d$. We consider a teacher-student scenario where the teacher has the same structure as the student with a hidden layer of smaller width $m^*\le m$. We describe how the empirical loss landscape is affected by the number $n$ of data samples and the width $m^*$ of the teacher network. In particular we determine how the probability that there be no spurious minima on the empirical loss depends on $n$, $d$, and $m^*$, thereby establishing conditions under which the neural network can in principle recover the teacher. We also show that under the same conditions gradient descent dynamics on the empirical loss converges and leads to small generalization error, i.e. it enables recovery in practice. Finally we characterize the time-convergence rate of gradient descent in the limit of a large number of samples. These results are confirmed by numerical experiments.

Motivation & Objective

• To understand when over-parametrized shallow neural networks with quadratic activation can recover the true teacher network via optimization.
• To identify the minimal number of training samples required for the empirical loss landscape to have a unique global minimum at the teacher network.
• To characterize the convergence dynamics of gradient descent on the empirical loss and its generalization performance.
• To quantify the nonasymptotic convergence rate of gradient descent in the large-sample limit.
• To explore the loss landscape structure using the string method to identify minimum energy paths between initial and teacher weights.

Proposed method

• Analyzes the empirical loss landscape in a teacher-student framework with one-hidden-layer networks and quadratic activation.
• Derives a critical sample complexity threshold $\alpha_c = m^* + 1$ for the number of samples $n$ per input dimension $d$, above which the minimizer set reduces to the teacher network with high probability.
• Models the evolution of eigenvalues of the Hessian-like matrix governing the dynamics using a system of ODEs resembling Lotka-Volterra equations.
• Uses gradient descent flow on the empirical loss to study convergence to low generalization error, with explicit bounds on the loss decay rate.
• Applies the string method to probe the loss landscape and identify minimum energy paths connecting initial weights to the teacher network.
• Employs asymptotic and numerical analysis to validate convergence rates and eigenvalue dynamics in high-dimensional limits.

Experimental results

Research questions

• RQ1 Under what conditions on the number of samples $n$, input dimension $d$, and teacher width $m^*$ does the empirical loss landscape have no spurious minima?
• RQ2 Can gradient descent on the empirical loss converge to the true teacher network, and under what sample complexity threshold does this occur?
• RQ3 What is the nonasymptotic convergence rate of gradient descent on the empirical loss in the large-sample regime?
• RQ4 How does the loss decay rate depend on model hyperparameters such as $d$, $m$, and $m^*$?
• RQ5 What structural features of the loss landscape—such as energy barriers or flat regions—can be revealed by the string method?

Key findings

• The empirical loss has zero value at all minimizers, but the set of minimizers reduces to the singleton containing only the teacher network with high probability when $n/d \geq m^* + 1$.
• Gradient descent on the empirical loss converges to a solution with small generalization error when $n/d \geq m^* + 1$, enabling practical recovery of the teacher.
• The loss decays as $O(t^{-2})$ for large $t$ in one regime and exponentially in another, depending on the relative values of $d$ and $m^*$.
• The convergence rate is bounded above by $C_1 / (1 + C_2 t)$ for some $C_1, C_2 > 0$, with explicit time constants derived from eigenvalue dynamics.
• The string method reveals that the loss landscape contains paths connecting initial weights to the teacher, including routes that traverse flat regions or energy barriers.
• Numerical validation confirms that the derived eigenvalue dynamics and convergence approximations are highly accurate, especially in high-dimensional limits.