[Paper Review] Acceleration with a Ball Optimization Oracle

Yair Carmon, Arun Jambulapati+5 more|arXiv (Cornell University)|Mar 18, 2020

Sparse and Compressive Sensing Techniques21 references4 citations

TL;DR

This paper introduces an accelerated algorithm for convex optimization using a ball optimization oracle, which returns the minimizer of a function within an ℓ₂ ball of radius r around a query point. The authors achieve a query complexity of O(r^{-2/3} log(1/ε)), improving upon the standard O(r^{-1} log(1/ε)) bound, and provide a matching lower bound, demonstrating optimality. The method enables faster convergence for regression problems like logistic, ℓ∞, and ℓp regression by efficiently implementing the oracle under Hessian stability.

ABSTRACT

Consider an oracle which takes a point $x$ and returns the minimizer of a convex function $f$ in an $\ell_2$ ball of radius $r$ around $x$. It is straightforward to show that roughly $r^{-1}\log\frac{1}ε$ calls to the oracle suffice to find an $ε$-approximate minimizer of $f$ in an $\ell_2$ unit ball. Perhaps surprisingly, this is not optimal: we design an accelerated algorithm which attains an $ε$-approximate minimizer with roughly $r^{-2/3} \log \frac{1}ε$ oracle queries, and give a matching lower bound. Further, we implement ball optimization oracles for functions with locally stable Hessians using a variant of Newton's method. The resulting algorithm applies to a number of problems of practical and theoretical import, improving upon previous results for logistic and $\ell_\infty$ regression and achieving guarantees comparable to the state-of-the-art for $\ell_p$ regression.

Motivation & Objective

• To improve the query complexity of first-order optimization methods that rely on a ball optimization oracle, which returns the minimizer of a function within an ℓ₂ ball of radius r around a given point.
• To close the gap between the standard O(r^{-1}) query complexity and the theoretical lower bound, showing that O(r^{-2/3}) is optimal.
• To implement the ball optimization oracle efficiently for functions with locally stable Hessians using a modified Newton method.
• To apply the accelerated framework to practical regression problems, including logistic, ℓ∞, and ℓp regression, achieving state-of-the-art or improved convergence rates.

Proposed method

• The authors use an accelerated variant of the Monteiro-Svaiter (MS) oracle framework, which relies on solving a proximal subproblem with a dynamically adjusted regularization parameter λ.
• They implement the MS oracle using binary search over λ to approximate the solution to the implicit equation λ = λ_r(y(λ)), enabling efficient access to the ball optimization oracle.
• The key innovation lies in relating convergence speed to the distance ‖x_{k+1} - y_k‖, which is bounded by r for all but possibly the last iteration, enabling the (R/r)^{2/3} complexity.
• For functions with locally stable Hessians—where the Hessian varies by at most a constant factor within an r-ball—the authors implement the ball oracle via Nesterov's accelerated gradient method in a Hessian-weighted norm.
• The method leverages a robust zero-chain property of a canonical hard function (Nemirovski’s function) to prove a matching Ω((R/r)^{2/3}) lower bound for any algorithm using an r-local oracle.
• The analysis uses Yao’s minimax principle and concentration inequalities on random projections to bound the probability of escaping the r-ball, establishing the lower bound under worst-case input distributions.

Experimental results

Research questions

• RQ1 Can the query complexity of first-order methods using a ball optimization oracle be improved beyond O(r^{-1})?
• RQ2 Is the O(r^{-2/3}) query complexity achievable and optimal for minimizing smooth convex functions via such oracles?
• RQ3 Can the ball optimization oracle be efficiently implemented for functions with locally stable Hessians?
• RQ4 How does the accelerated framework perform on practical regression problems such as logistic and ℓp regression?
• RQ5 What is the information-theoretic limit of optimization when only local oracle access (within an r-ball) is available?

Key findings

• The proposed algorithm achieves an O(r^{-2/3} log(1/ε)) query complexity for finding an ε-approximate minimizer, significantly improving over the standard O(r^{-1} log(1/ε)) bound.
• A matching lower bound of Ω((R/r)^{2/3}) is proven for any algorithm using an r-local oracle, establishing the optimality of the proposed complexity.
• The method enables faster convergence for logistic regression, ℓ∞ regression, and ℓp regression, matching or improving upon state-of-the-art rates.
• The ball optimization oracle is efficiently implemented for functions with locally stable Hessians using a Hessian-weighted accelerated gradient method.
• The analysis shows that the convergence rate depends on the squared distance between the MS oracle’s intermediate points y_k and x_{k+1}, which is bounded by r, enabling the improved complexity.
• The lower bound holds even for an r-local oracle that reveals all function values within the r-ball, not just the minimizer, demonstrating the robustness of the complexity limit.