[Paper Review] Characterizing the implicit bias via a primal-dual analysis

Ziwei Ji, Matus Telgarsky|arXiv (Cornell University)|Jun 11, 2019

Stochastic Gradient Optimization Techniques33 references15 citations

TL;DR

This paper provides a primal-dual analysis of gradient descent on linearly separable data, showing that its implicit bias is exactly characterized by the solution to a dual optimization problem with a smoothed margin. For the exponential loss, it establishes tight $O( ext{ln}(n)/ ext{ln}(t))$ and $O( ext{ln}(n)/t)$ convergence rates to the $ℓ_2$ maximum margin direction under constant and aggressive step sizes, respectively, improving upon prior $\widetilde{O}(1/\sqrt{t})$ and suboptimal $O(1/t)$ rates.

ABSTRACT

This paper shows that the implicit bias of gradient descent on linearly separable data is exactly characterized by the optimal solution of a dual optimization problem given by a smoothed margin, even for general losses. This is in contrast to prior results, which are often tailored to exponentially-tailed losses. For the exponential loss specifically, with $n$ training examples and $t$ gradient descent steps, our dual analysis further allows us to prove an $O(\ln(n)/\ln(t))$ convergence rate to the $\ell_2$ maximum margin direction, when a constant step size is used. This rate is tight in both $n$ and $t$, which has not been presented by prior work. On the other hand, with a properly chosen but aggressive step size schedule, we prove $O(1/t)$ rates for both $\ell_2$ margin maximization and implicit bias, whereas prior work (including all first-order methods for the general hard-margin linear SVM problem) proved $\widetilde{O}(1/\sqrt{t})$ margin rates, or $O(1/t)$ margin rates to a suboptimal margin, with an implied (slower) bias rate. Our key observations include that gradient descent on the primal variable naturally induces a mirror descent update on the dual variable, and that the dual objective in this setting is smooth enough to give a faster rate.

Motivation & Objective

• To characterize the implicit bias of gradient descent on linearly separable data for a broad class of losses, including non-exponentially-tailed ones.
• To establish tight convergence rates for directional convergence to the $\ell_2$ maximum margin direction under various step size schedules.
• To demonstrate that gradient descent on the primal induces mirror descent on the dual, enabling faster convergence via dual smoothness.
• To resolve open questions about the optimality of margin maximization rates in first-order methods for the hard-margin linear SVM problem.

Proposed method

• Formalizing the implicit bias via a dual optimization problem defined by a smoothed margin function.
• Analyzing gradient descent as inducing a mirror descent update on the dual variable, leveraging the smoothness of the dual objective.
• Deriving convergence rates by bounding the Bregman divergence between dual iterates and the optimal dual solution.
• Using a primal-dual Lyapunov function to track progress and establish $O(1/t)$ and $O(\text{ln}(n)/\text{ln}(t))$ rates.
• Applying the analysis to both constant and aggressive step size schedules to achieve tight rates.
• Proving tightness of the rates through comparison with prior work and known lower bounds.

Experimental results

Research questions

• RQ1 What is the exact characterization of the implicit bias of gradient descent on linearly separable data for general losses beyond exponentially-tailed ones?
• RQ2 Can tighter convergence rates be proven for directional convergence to the $\ell_2$ maximum margin direction compared to prior $\widetilde{O}(1/\sqrt{t})$ bounds?
• RQ3 Does an aggressive step size schedule enable $O(1/t)$ convergence to both the maximum margin and the implicit bias, and is this rate tight?
• RQ4 How does the primal-dual perspective reveal faster convergence than standard primal analysis?
• RQ5 Can the dual smoothness be exploited to achieve faster convergence than previously known $O(1/\sqrt{t})$ rates?

Key findings

• For general losses asymptoting to zero, the normalized gradient descent iterates converge to the solution of a dual problem with a smoothed margin, at an $O(1/\sqrt{t})$ rate.
• With a constant step size, the directional convergence to the $\ell_2$ maximum margin direction achieves a tight $O(\text{ln}(n)/\text{ln}(t))$ rate, improving upon the prior $O(1/\text{ln}(t))$ rate without explicit $n$-dependence.
• For the exponential loss, an aggressive step size schedule yields $O(\text{ln}(n)/t)$ convergence rates to both the $\ell_2$ margin and the implicit bias, which are tight and improve upon prior $\widetilde{O}(1/\sqrt{t})$ and suboptimal $O(1/t)$ rates.
• The primal-dual analysis reveals that gradient descent on the primal induces mirror descent on the dual, and the smoothness of the dual objective enables faster $O(1/t)$ convergence rates.
• The analysis proves that the $O(\text{ln}(n)/t)$ rate is tight in both $n$ and $t$, resolving an open question about optimality in first-order methods for hard-margin linear SVM.
• The results hold for a broad class of losses, including polynomially-tailed and exponentially-tailed losses, extending beyond prior work restricted to exponential tails.