[Paper Review] Characterizing Implicit Bias in Terms of Optimization Geometry

Suriya Gunasekar, Jason Lee+2 more|arXiv (Cornell University)|Feb 22, 2018

Sparse and Compressive Sensing Techniques31 references47 citations

TL;DR

The paper analyzes how different optimization algorithms bias the selection of global minima in underdetermined linear models, linking implicit bias to optimization geometry (potentials, norms) across losses with unique finite roots and strictly monotone losses.

ABSTRACT

We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or separable linear classification problems. We explore the question of whether the specific global minimum (among the many possible global minima) reached by an algorithm can be characterized in terms of the potential or norm of the optimization geometry, and independently of hyperparameter choices such as step-size and momentum.

Motivation & Objective

• Determine how the implicit bias of generic optimization methods selecting among many global minima can be characterized by optimization geometry and initialization.
• Distinguish how losses with unique finite roots versus strictly monotone losses influence the limit behavior of various algorithms.
• Provide precise bias characterizations for mirror descent, natural gradient descent, and steepest descent under different geometries.

Proposed method

• Analyze linear models under empirical risk minimization with losses having unique finite roots (e.g., squared loss) and strictly monotone losses (e.g., logistic, exponential).
• Characterize the limit point or limit direction of optimization trajectories (w_infty or w_bar_infty) under different algorithms and geometries.
• Derive theorems for mirror descent (and constrained variants) showing limit points as minimizers of D_psi within the feasible set.
• Show infinitesimal-step results for natural gradient descent aligning with mirror descent in the limit when psi is quadratic; otherwise, finite step-sizes may alter bias.
• Establish that for strictly monotone losses, gradient descent and steepest descent converge in direction to max-margin solutions, independent of initialization and often step-size (under certain conditions).
• Investigate AdaGrad’s bias showing dependence on initialization and initial G_t, even for monotone losses.

Experimental results

Research questions

• RQ1 Can the implicit bias of various optimization algorithms be characterized solely by the optimization geometry (potential or norm) independent of step-size, momentum, or stochasticity?
• RQ2 How do losses with unique finite roots versus strictly monotone losses affect the limit behavior (point vs. direction) of optimization trajectories under different geometries?
• RQ3 What are the limit points or directions for mirror descent, natural gradient descent, and steepest descent with respect to different potentials/norms?
• RQ4 Does infinitesimal-step analysis recover bias characterizations for non-Euclidean geometries, and how do finite steps alter these biases?
• RQ5 How do adaptive methods like AdaGrad influence the implicit bias across loss types?

Key findings

• For losses with a unique finite root, mirror descent converges to the global minimizer that minimizes the Bregman divergence D_psi to the initialization (and to the minimum-psi global minimizer when starting at the psi-minimizer).
• Dual-space momentum in mirror descent preserves the data-manifold constraint and preserves the bias toward the psi-minimizer under suitable conditions; primal momentum can shift the limit point away from this bias.
• Natural gradient descent with infinitesimal step-size shares the same limit as mirror descent, but with finite step-sizes the bias generally differs and can depend on the step-sizes.
• Steepest descent with generic norms does not generally admit a simple, initialization-independent bias for finite or infinitesimal steps, unlike the Euclidean case.
• For strictly monotone losses (e.g., exponential/logistic), gradient descent and steepest descent converge in direction to the maximum margin separator with respect to the chosen norm, independent of initialization and (under constraints) small step-size.
• AdaGrad’s implicit bias depends on initialization and initial G_t, even for monotone losses, due to bounded growth of G_t.
• Matrix factorization parameterization (W = UV^T) introduces additional non-convexity and ties into the broader discussion of implicit bias under different optimization schemes.