[Paper Review] Gradient Descent Maximizes the Margin of Homogeneous Neural Networks

Kaifeng Lyu, Jian Li|arXiv (Cornell University)|Jun 13, 2019

Stochastic Gradient Optimization TechniquesComputer Science70 references57 citations

TL;DR

The paper shows that gradient descent/flow on homogeneous neural networks implicitly maximizes the normalized margin, with a monotone smoothed margin that converges to a KKT point of a max-margin problem, and provides asymptotic loss/weight growth rates evidenced on experiments.

ABSTRACT

In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can define a smoothed version of the normalized margin which increases over time. We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem. Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks. We conduct several experiments to justify our theoretical finding on MNIST and CIFAR-10 datasets. Finally, as margin is closely related to robustness, we discuss potential benefits of training longer for improving the robustness of the model.

Motivation & Objective

• Investigate the implicit regularization of gradient descent/flow in homogeneous neural networks (ReLU/LeakyReLU, with or without biases).
• Show that a smoothed normalized margin increases over time and converges to a max-margin objective under natural assumptions.
• Characterize convergence properties, rate of loss decay, weight growth, and relation to margin-based optimality (KKT points).
• Provide empirical validation on MNIST and CIFAR-10 and discuss robustness implications of longer training.

Proposed method

• Analyze gradient flow and gradient descent for homogeneous networks under exponential-type losses (including cross-entropy/logistic).
• Define and study the normalized margin ar{b3} via q_min( )/||btheta||^L and its smoothed version tilde{b3} using LogSumExp.
• Prove that tilde{b3} (and similarly hat{b3} for gradient descent) is non-decreasing after a training time t0 and that L(btheta(t)) 0a 0 while ||btheta(t)|| grows unbounded.
• Show that limit points of normalized parameter directions align with KKT points of a margin-maximization problem (P).
• Derive tight asymptotic rates for loss decay and weight growth, and relate to max-margin SVM with NTK kernel at limit points.
• Extend results to broader loss classes (logistic, cross-entropy, exponential tails) and multi-homogeneous networks.

Experimental results

Research questions

• RQ1 Does gradient descent/flow on homogeneous networks bias the solution toward max-margin directions?
• RQ2 Can a smoothed, normalized margin be shown to be non-decreasing, and does it converge to a margin optimality condition?
• RQ3 What is the asymptotic behavior of training loss and weight norms under these dynamics, and to what extent do limit points satisfy KKT conditions for a margin-maximization problem?
• RQ4 How do these theoretical results extend to broader loss functions and multi-homogeneous architectures?
• RQ5 Do empirical experiments support the margin-maximization behavior and potential robustness benefits of longer training?

Key findings

• The normalized margin is not globally monotone, but there exists a smoothed version that is non-decreasing after a time t0 and converges to the true normalized margin as training grows.
• Under the stated assumptions, training loss tends to zero and the weight norm grows without bound, with the smoothed margin providing a tight approximation to the actual margin.
• Any limit point of the direction of the parameters is aligned with a KKT point of a margin-maximization problem, indicating implicit margin maximization by gradient dynamics.
• The approach extends to a broad class of losses with exponential tails (including logistic and cross-entropy) and to multi-homogeneous networks, preserving the margin-maximization behavior.
• Empirical results on MNIST and CIFAR-10 show that longer training can increase the normalized margin and improve L2-robustness, especially with loss-based learning-rate schedules.
• Theoretical results generalize prior work on linear models to deep homogeneous networks and provide explicit convergence rates for loss and weight growth.