[Paper Review] Piecewise Strong Convexity of Neural Networks

Tristan Milne|arXiv (Cornell University)|Oct 30, 2018

Stochastic Gradient Optimization Techniques21 references4 citations

TL;DR

This paper establishes that the regularized loss function of a ReLU neural network is piecewise strongly convex on an important open set in weight space, ensuring all differentiable critical points in this region are local minima and local minima are isolated. The result holds without distributional assumptions on data and is validated empirically on image classification tasks, showing SGD trajectories remain nearly always in the piecewise strongly convex regime.

ABSTRACT

We study the loss surface of a feed-forward neural network with ReLU non-linearities, regularized with weight decay. We show that the regularized loss function is piecewise strongly convex on an important open set which contains, under some conditions, all of its global minimizers. This is used to prove that local minima of the regularized loss function in this set are isolated, and that every differentiable critical point in this set is a local minimum, partially addressing an open problem given at the Conference on Learning Theory (COLT) 2015; our result is also applied to linear neural networks to show that with weight decay regularization, there are no non-zero critical points in a norm ball obtaining training error below a given threshold. We also include an experimental section where we validate our theoretical work and show that the regularized loss function is almost always piecewise strongly convex when restricted to stochastic gradient descent trajectories for three standard image classification problems.

Motivation & Objective

• To understand the geometry of the loss surface of ReLU neural networks with weight decay regularization.
• To address the open problem from COLT 2015 on the nature of critical points in non-convex deep learning optimization.
• To establish conditions under which local minima are isolated and all differentiable critical points are local minima.
• To demonstrate empirically that SGD trajectories remain in the piecewise strongly convex regime for standard image classification networks.
• To provide a design principle for neural networks by characterizing the set where piecewise strong convexity is guaranteed.

Proposed method

• Analyzes the Hessian of the regularized loss function to establish piecewise strong convexity based on ReLU activation patterns.
• Identifies an open set in weight space—containing the origin and all global minimizers under certain conditions—where the loss is piecewise strongly convex.
• Uses subgradient and second-order analysis to prove that all differentiable critical points in this set are local minima.
• Applies the framework to linear networks with weight decay, proving no non-zero critical points exist in a norm ball with low training error.
• Empirically validates the theory by computing the normalized second derivative along SGD trajectories on MNIST, CIFAR10, and CIFAR100.
• Uses a threshold-based criterion to assess when the loss function is piecewise strongly convex, based on the loss value and second derivative behavior.

Experimental results

Research questions

• RQ1 Under what conditions is the regularized loss function of a ReLU neural network piecewise strongly convex?
• RQ2 Are all differentiable critical points in the piecewise strongly convex region local minima?
• RQ3 Can local minima of the regularized loss function be isolated under the same conditions?
• RQ4 Does the loss function remain in a piecewise strongly convex regime along stochastic gradient descent trajectories in practice?
• RQ5 Can the absence of non-zero critical points in linear networks with weight decay be rigorously proven under a training error threshold?

Key findings

• The regularized loss function is piecewise strongly convex on an open set containing the origin and, under conditions, all global minimizers.
• All differentiable critical points within this open set are local minima, and local minima are isolated.
• For linear networks with weight decay, no non-zero critical points exist in a norm ball where training error is below a given threshold.
• Empirical results show that over 95% of SGD trajectories on MNIST, CIFAR10, and CIFAR100 remain in the piecewise strongly convex regime, with normalized second derivatives consistently above 10.
• The normalized second derivative remains large (often >10) for most of the training process, indicating strong convexity behavior along trajectories.
• The results hold without assumptions on the data distribution, making them broadly applicable to real-world deep learning scenarios.