[Paper Review] A geometric alternative to Nesterov's accelerated gradient descent

Sébastien Bubeck, Yin Tat Lee+1 more|arXiv (Cornell University)|Jun 26, 2015

Stochastic Gradient Optimization TechniquesComputer Science3 references91 citations

TL;DR

This paper proposes a geometric first-order optimization method, Geometric Descent (GeoD), that achieves the optimal convergence rate of Nesterov's accelerated gradient descent through a novel geometric interpretation involving shrinking enclosing balls. By combining gradient information with ellipsoid-like updates via line searches and ball intersections, GeoD matches the accelerated rate of $1 - \frac{1}{\sqrt{\kappa}}$ per iteration while offering clearer intuition than Nesterov's method, with empirical results showing competitive performance on classification tasks and worst-case problems.

ABSTRACT

We propose a new method for unconstrained optimization of a smooth and strongly convex function, which attains the optimal rate of convergence of Nesterov's accelerated gradient descent. The new algorithm has a simple geometric interpretation, loosely inspired by the ellipsoid method. We provide some numerical evidence that the new method can be superior to Nesterov's accelerated gradient descent.

Motivation & Objective

• To develop a first-order optimization method with the optimal convergence rate of $1 - \frac{1}{\sqrt{\kappa}}$ for smooth, strongly convex functions.
• To provide a geometrically intuitive alternative to Nesterov's accelerated gradient descent, which is known for its complex and opaque derivation.
• To improve interpretability of acceleration in first-order methods by leveraging ball enclosures and intersection geometry.
• To empirically evaluate the new method against established methods like AFG, AFGwR, L-BFGS, and steepest descent on classification and worst-case problems.

Proposed method

• The method maintains two balls: one centered at the gradient step $x^{++} = x - \frac{1}{\alpha}\nabla f(x)$, and another from previous iterations, both enclosing the optimum.
• At each iteration, the algorithm computes the minimum enclosing ball of the intersection of two balls: one derived from current gradient information and one from prior iterates.
• It uses line searches to compute key points: $x^+ = \text{line\_search}(x, x - \nabla f(x))$ and $x^{++} = \text{line\_search}(x, x - \frac{1}{\alpha}\nabla f(x))$.
• The radius of the enclosing ball shrinks at rate $1 - \frac{1}{\sqrt{\kappa}}$, matching Nesterov’s optimal rate, by leveraging both current and historical gradient data.
• The algorithm performs two line searches per iteration: one along the gradient direction and one to compute the new iterate via intersection of enclosing balls.
• The method is formally defined in Algorithm 2 (GeoD), which iteratively updates the center and radius of the smallest enclosing ball of the intersection of two shrinking balls.

Experimental results

Research questions

• RQ1 Can a geometric interpretation of first-order optimization be constructed that achieves the same convergence rate as Nesterov’s accelerated gradient descent?
• RQ2 Does combining gradient descent with ellipsoid-like ball intersection principles lead to a more interpretable and effective acceleration mechanism?
• RQ3 Can the new method outperform Nesterov’s method and other first-order methods in practice, especially in terms of convergence speed and robustness?
• RQ4 How does the performance of the geometric method scale on worst-case problems with known lower bounds on convergence rates?

Key findings

• GeoD achieves the optimal convergence rate of $1 - \frac{1}{\sqrt{\kappa}}$ per iteration, matching Nesterov’s accelerated gradient descent.
• In binary classification experiments on 40 LIBSVM datasets, GeoD outperforms steepest descent, AFG, and AFGwR, though it is outperformed by L-BFGS.
• On a worst-case problem with condition number $\kappa = \beta$, GeoD and AFGwR converge faster than steepest descent after $\Theta(n)$ iterations, despite using only one unit of memory.
• The method demonstrates that acceleration can be understood through the shrinking of the intersection of two balls—one from current gradient and one from prior iterates—offering a clearer geometric intuition than Nesterov’s method.
• The algorithm requires only one gradient evaluation and two line searches per iteration, maintaining computational efficiency while achieving optimal convergence.
• Numerical results suggest that GeoD can be competitive with L-BFGS when extended to use multiple ball intersections, indicating potential for further performance gains with memory.