[Paper Review] The Implicit Bias of Gradient Descent on Separable Data

Daniel Soudry, Elad Hoffer+3 more|arXiv (Cornell University)|Oct 27, 2017

Stochastic Gradient Optimization Techniques15 citations

TL;DR

This paper demonstrates that gradient descent on unregularized logistic regression with linearly separable data implicitly converges to the maximum margin (L2) solution, even without explicit regularization. The convergence is slow, scaling as O(1/log t), explaining why training continues to improve generalization beyond zero training error.

ABSTRACT

We examine gradient descent on unregularized logistic regression problems, with homogeneous linear predictors on linearly separable datasets. We show the predictor converges to the direction of the max-margin (hard margin SVM) solution. The result also generalizes to other monotone decreasing loss functions with an infimum at infinity, to multi-class problems, and to training a weight layer in a deep network in a certain restricted setting. Furthermore, we show this convergence is very slow, and only logarithmic in the convergence of the loss itself. This can help explain the benefit of continuing to optimize the logistic or cross-entropy loss even after the training error is zero and the training loss is extremely small, and, as we show, even if the validation loss increases. Our methodology can also aid in understanding implicit regularization n more complex models and with other optimization methods.

Motivation & Objective

• To understand the implicit bias of gradient descent in unregularized logistic regression on linearly separable datasets.
• To determine whether gradient descent converges to the maximum margin solution despite the absence of explicit regularization.
• To characterize the convergence rate of the normalized weight vector toward the max-margin direction.
• To extend the analysis to monotone decreasing loss functions with exponential tails and multi-class settings.
• To investigate the implications for deep learning generalization and the role of optimization dynamics.

Proposed method

• Analyzes gradient descent on unregularized logistic regression with homogeneous linear predictors on linearly separable data.
• Proves that the normalized weight vector w(t)/||w(t)|| converges to the L2 maximum margin separator (hard margin SVM solution).
• Uses asymptotic analysis and differential equation approximations to study the dynamics of the normalized direction.
• Establishes convergence rates via bounding the decay of the distance to the max-margin solution.
• Extends results to smooth, monotone decreasing, lower-bounded losses with exponential tails and multi-class problems.
• Considers a restricted setting for training a single weight layer in deep networks.

Experimental results

Research questions

• RQ1 Does gradient descent on unregularized logistic regression with linearly separable data converge to the maximum margin solution?
• RQ2 What is the rate of convergence of the normalized weight vector to the max-margin direction?
• RQ3 How does the implicit bias of gradient descent compare to other optimization methods like ADAM?
• RQ4 Can the implicit bias be generalized to other loss functions beyond logistic loss?
• RQ5 Why does training continue to improve generalization even after training loss is nearly zero?

Key findings

• Gradient descent on unregularized logistic regression with linearly separable data converges to the L2 maximum margin solution, equivalent to the hard margin SVM.
• The convergence rate of the normalized weight vector toward the max-margin direction is O(1/log t), which is extremely slow.
• For some degenerate datasets, the convergence rate slows further to O(log log t / log t).
• The implicit bias is specific to gradient descent; adaptive methods like ADAM do not exhibit the same behavior.
• The slow convergence explains why generalization continues to improve even after training loss is nearly zero and validation loss may increase.
• The results generalize to other monotone decreasing, lower-bounded losses with exponential tails and to multi-class settings.