[Paper Review] Primal-dual subgradient methods for minimizing uniformly convex functions

Anatoli B. Juditsky, Yuri Nesterov|arXiv (Cornell University)|Jan 8, 2014

Stochastic Gradient Optimization TechniquesComputer Science14 references59 citations

TL;DR

This paper develops primal-dual subgradient methods for minimizing uniformly convex functions in non-Euclidean settings, achieving minimax optimal convergence rates that match the best-known bounds up to logarithmic factors in the number of iterations. The approach adapts automatically to unknown strong or uniform convexity parameters, ensuring optimal performance without prior knowledge of the objective's curvature properties.

ABSTRACT

We discuss non-Euclidean deterministic and stochastic algorithms for optimization problems with strongly and uniformly convex objectives. We provide accuracy bounds for the performance of these algorithms and design methods which are adaptive with respect to the parameters of strong or uniform convexity of the objective: in the case when the total number of iterations $N$ is fixed, their accuracy coincides, up to a logarithmic in $N$ factor with the accuracy of optimal algorithms.

Motivation & Objective

• To design deterministic and stochastic first-order algorithms for large-scale non-Euclidean optimization with uniformly convex objectives.
• To achieve minimax optimal convergence rates for uniformly convex functions, matching known lower bounds up to logarithmic factors in the number of iterations.
• To develop adaptive methods that do not require prior knowledge of the strong or uniform convexity parameters of the objective function.
• To extend non-Euclidean first-order methods beyond strongly convex to uniformly convex settings with general convexity parameters $\rho \in [2, \infty)$.
• To provide efficient solution schemes for the proximal subproblems arising in the primal-dual framework, particularly for the simplex and hyperoctahedron constraint sets.

Proposed method

• The method employs a primal-dual subgradient framework based on a non-Euclidean proximal setup, using a distance-generating function $d(x)$ with a Bregman divergence to define the proximal term.
• It introduces a dualized formulation of the subproblem (11) via Lagrangian relaxation of coupling constraints, enabling decomposition into independent 2D subproblems.
• For the standard simplex and hyperoctahedron, the dual problem is solved by reducing each subproblem to minimizing a function of the form $su + tv + u\ln u + v\ln v$ under linear constraints.
• The solution to each 2D subproblem is computed by checking whether the equality-constrained minimizer satisfies optimality conditions; otherwise, the bound constraints are active and the solution is set accordingly.
• The algorithm adapts to unknown convexity parameters $\rho$ and $\mu$ by using a self-adaptive step-size rule that ensures convergence without tuning.
• The method achieves convergence rates of order $O\left(\mu^{-2/\rho} \epsilon^{-(2(\rho-1)/\rho)}\right)$, matching known lower bounds up to logarithmic factors.

Experimental results

Research questions

• RQ1 Can primal-dual subgradient methods achieve minimax optimal convergence rates for uniformly convex functions in non-Euclidean spaces?
• RQ2 How can such methods be made adaptive to unknown strong or uniform convexity parameters $\mu$ and $\rho$?
• RQ3 What is the computational complexity of solving the proximal subproblems in the primal-dual framework for standard constraint sets like the simplex and hyperoctahedron?
• RQ4 Can the convergence rate of first-order methods for uniformly convex problems be improved beyond the standard $O(\epsilon^{-2})$ bound of general convex functions?
• RQ5 Under what conditions does the non-Euclidean framework offer advantages over the Euclidean one in terms of conditioning and convergence speed?

Key findings

• The proposed primal-dual subgradient method achieves a convergence rate of $O\left(\mu^{-2/\rho} \epsilon^{-(2(\rho-1)/\rho)}\right)$ for uniformly convex functions with parameters $\rho \in [2, \infty)$ and $\mu \geq 0$, matching known lower bounds up to logarithmic factors.
• The method is adaptive: it achieves optimal performance without requiring prior knowledge of the convexity parameters $\mu$ and $\rho$, making it suitable for black-box optimization.
• For the standard simplex and hyperoctahedron constraint sets, the proximal subproblems can be solved efficiently by reducing them to 2D optimization problems with closed-form solutions or simple root-finding procedures.
• The method retains optimality in the non-Euclidean setting even when the condition number $\lambda = \mathcal{L}(f)/\mu(f)$ is large, unlike Euclidean methods that may degrade under such conditions.
• The analysis confirms that the worst-case complexity of the method is optimal in the sense of minimax theory, with no faster rate possible for the given class of problems.
• The results extend prior work on strongly convex (i.e., $\rho=2$) and smooth uniformly convex problems to the general uniformly convex case with arbitrary $\rho \geq 2$.