[Paper Review] Avoiding Spurious Local Minima in Deep Quadratic Networks

Abbas Kazemipour, Brett W. Larsen+1 more|arXiv (Cornell University)|Dec 31, 2019

Stochastic Gradient Optimization Techniques38 references4 citations

TL;DR

This paper establishes that deep quadratic neural networks avoid spurious local minima in their loss landscape when overparameterized, proving gradient descent converges to a global minimum with probability one under mild conditions. It shows that adding the input norm as a regressor or using structured regularization enables escape from spurious stationary points, even in two-layer networks with as few as d neurons for d-dimensional inputs.

ABSTRACT

Despite their practical success, a theoretical understanding of the loss landscape of neural networks has proven challenging due to the high-dimensional, non-convex, and highly nonlinear structure of such models. In this paper, we characterize the training landscape of the mean squared error loss for neural networks with quadratic activation functions. We prove existence of spurious local minima and saddle points which can be escaped easily with probability one when the number of neurons is greater than or equal to the input dimension and the norm of the training samples is used as a regressor. We prove that deep overparameterized neural networks with quadratic activations benefit from similar nice landscape properties. Our theoretical results are independent of data distribution and fill the existing gap in theory for two-layer quadratic neural networks. Finally, we empirically demonstrate convergence to a global minimum for these problems.

Motivation & Objective

• To theoretically characterize the loss landscape of deep quadratic neural networks with mean squared error loss.
• To resolve the gap in existing theory by proving global convergence for two-layer quadratic networks with k ≥ d neurons, independent of data distribution.
• To demonstrate that overparameterization and specific regularization strategies eliminate spurious local minima and saddle points.
• To establish conditions under which gradient descent converges to a global minimizer in quadratic networks, even in high-dimensional, non-convex settings.

Proposed method

• Analyzes the mean squared error loss function in two-layer and deep quadratic neural networks with ReLU-like quadratic activations.
• Introduces data normalization by adding the L2 norm of input samples as an explicit regressor to escape spurious local minima.
• Applies regularization via a Frobenius norm penalty on the factorization matrix to enforce full-rank solutions and avoid low-rank traps.
• Uses eigendecomposition and tensor decomposition theory to reframe the optimization as a convex problem in the weight space under full-rank conditions.
• Employs gradient descent with arbitrary step size, proving convergence to global minimum when the network is overparameterized and initialized properly.
• Generalizes results to higher-order polynomial networks (PL networks) by showing that with k = binom(d+p-1, d-1) neurons and proper initialization, global convergence is guaranteed.

Experimental results

Research questions

• RQ1 Under what conditions do two-layer quadratic neural networks avoid spurious local minima in their loss landscape?
• RQ2 Can gradient descent converge to a global minimum in quadratic networks when the number of neurons is as low as the input dimension?
• RQ3 How does data normalization—specifically adding the input norm as a regressor—affect the escape from spurious stationary points?
• RQ4 What role does regularization play in eliminating spurious solutions in symmetric matrix factorizations of quadratic networks?
• RQ5 Do deep overparameterized quadratic networks inherit the favorable optimization properties of their two-layer counterparts?

Key findings

• For two-layer quadratic networks with k ≥ d neurons, gradient descent escapes spurious local minima and saddle points with probability one when the input norm is used as a regressor.
• Theoretical results hold independently of the data distribution, removing restrictive assumptions present in prior work.
• Overparameterized deep quadratic networks exhibit no spurious stationary points, enabling global convergence via gradient descent.
• When k = d, training reduces to a least squares problem with quadratic features, solvable via eigendecomposition.
• For higher-order polynomial networks (p > 2), global convergence is guaranteed with k = binom(d+p-1, d-1) neurons and proper initialization.
• The regularization penalty used in the analysis ensures full-rank factorizations and avoids low-rank solutions that could lead to spurious minima.