[Paper Review] Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descent

Surbhi Goel, Aravind Gollakota+3 more|arXiv (Cornell University)|Jun 22, 2020

Stochastic Gradient Optimization Techniques27 references14 citations

TL;DR

This paper establishes the first superpolynomial lower bounds for learning one-layer neural networks using gradient descent under the Gaussian distribution. By constructing a family of neural networks that are exactly orthogonal under spherically symmetric distributions, the authors prove that gradient descent and statistical query algorithms require superpolynomial time to achieve low test error, even for ReLU and sigmoid activations, ruling out efficient learning in polynomial time for this setting.

ABSTRACT

We prove the first superpolynomial lower bounds for learning one-layer neural networks with respect to the Gaussian distribution using gradient descent. We show that any classifier trained using gradient descent with respect to square-loss will fail to achieve small test error in polynomial time given access to samples labeled by a one-layer neural network. For classification, we give a stronger result, namely that any statistical query (SQ) algorithm (including gradient descent) will fail to achieve small test error in polynomial time. Prior work held only for gradient descent run with small batch sizes, required sharp activations, and applied to specific classes of queries. Our lower bounds hold for broad classes of activations including ReLU and sigmoid. The core of our result relies on a novel construction of a simple family of neural networks that are exactly orthogonal with respect to all spherically symmetric distributions.

Motivation & Objective

• To establish unconditional lower bounds on the time required for gradient descent to learn one-layer neural networks under the Gaussian distribution.
• To demonstrate that even overparameterized networks trained with gradient descent fail to generalize efficiently in polynomial time.
• To extend these results to the statistical query (SQ) model, showing that any SQ algorithm using inner-product queries also requires superpolynomial time.
• To construct a family of neural networks that are exactly orthogonal under spherically symmetric distributions, enabling the lower bound construction.
• To rule out efficient learning via standard deep learning practices such as training a polynomial-size network with gradient descent on square or logistic loss.

Proposed method

• Construct a family of one-layer neural networks with ReLU or sigmoid activations that are pairwise exactly orthogonal under any spherically symmetric distribution, including the standard Gaussian.
• Use this orthogonality to build a concept class where any classifier must make at least $ n^{ ilde{ heta}( ext{poly}( ext{log}~m))} $ queries to achieve low error.
• Leverage the statistical query (SQ) model to show that any algorithm relying on inner-product queries—such as gradient descent—requires superpolynomial time.
• Prove that the gradient of the population loss can be approximated using statistical queries of tolerance $ au $, linking SQ lower bounds to gradient descent performance.
• Apply a general SQ lower bound framework based on the SDA (Statistical Query Dimension) and inner-product query complexity to derive the final bounds.
• Conduct experiments on overparameterized networks trained with gradient descent on both regression and classification tasks, showing high test error despite low training error.

Experimental results

Research questions

• RQ1 Can gradient descent efficiently learn one-layer neural networks with ReLU or sigmoid activations under the standard Gaussian distribution?
• RQ2 Do superpolynomial lower bounds hold for gradient descent even when the network is overparameterized or in the NTK regime?
• RQ3 Can statistical query algorithms, including those based on gradient descent, learn such networks in polynomial time?
• RQ4 Is there a fundamental limitation in learning one-layer networks due to the geometry of the data and function class, independent of optimization heuristics?
• RQ5 Do standard deep learning practices—training a polynomial-size network with gradient descent on square or logistic loss—fail to generalize efficiently in this setting?

Key findings

• The paper proves that any statistical query algorithm using inner-product queries requires $ n^{ ilde{ heta}( ext{poly}( ext{log}~m))} $ queries to learn one-layer ReLU or sigmoid networks under the standard Gaussian distribution.
• For classification tasks, learning a one-layer network with $ m $ hidden units to an advantage of $ rac{1}{2} - ilde{ heta}(m^{-b}) $ requires $ n^{ ilde{ heta}( ext{log}~m)} $ queries, ruling out polynomial-time learning.
• The lower bounds hold unconditionally and do not depend on the architecture of the classifier, only on the algorithm (gradient descent or SQ).
• The construction of a family of functions that are exactly orthogonal under spherically symmetric distributions is central to proving the lower bounds.
• Experiments confirm the theory: overparameterized networks achieve near-zero training error but maintain high test error, consistent with the superpolynomial lower bounds.
• The results extend to both regression (square-loss) and classification (sign of softmax output), showing that common training practices fail to generalize efficiently.