[Paper Review] Solving strongly convex-concave composite saddle point problems with a small dimension of one of the variables

Egor Gladin, Ilya Kuruzov+4 more|arXiv (Cornell University)|Oct 5, 2020

Heat Transfer and Mathematical Modeling4 citations

TL;DR

This paper proposes a novel algorithm for solving strongly convex-concave composite saddle point problems where one variable group has small dimension (up to ~100). It leverages a decomposition strategy: for the high-dimensional variable, it uses accelerated gradient methods with inexact gradient information; for the low-dimensional variable, it applies ellipsoid or Polyak's method. The key contribution is a complexity bound of $\widetilde{O}(\sqrt{L_{xx}/\mu_x + L_{xy}^2/(\mu_x\mu_y) + L_{yy}/\mu_y})$ for the general case, and $\widetilde{O}(\sqrt{L_{xx}/\mu_x + L_{yy}/\mu_y})$ for the case of very small dimension (up to 5) using a new multidimensional dichotomy method with inexact gradients.

ABSTRACT

The article is devoted to the development of algorithmic methods ensuring efficient complexity bounds for strongly convex-concave saddle point problems in the case when one of the groups of variables is high-dimensional, and the other is relatively low-dimensional (up to a hundred). The proposed technique is based on reducing problems of this type to a problem of minimizing a convex (maximizing a concave) functional in one of the variables, for which it is possible to find an approximate gradient at an arbitrary point with the required accuracy using an auxiliary optimization subproblem with another variable. In this case, the ellipsoid method is used for low-dimensional problems (if necessary, with an inexact $\\delta$-subgradient), and accelerated gradient methods are used for high-dimensional problems. For the case of a very small dimension of one of the groups of variables (up to 5), an approach based on a new version of the multidimensional analog of the Yu. E. Nesterov's method on the square (multidimensional dichotomy) is proposed with the possibility of using inexact values of the gradient of the objective functional.

Motivation & Objective

• To develop efficient algorithms for solving strongly convex-concave composite saddle point problems where one variable group has small dimension (up to ~100), while the other has high dimension.
• To address the challenge of achieving optimal or near-optimal convergence rates when gradient information is only available with inexactness or through subproblems.
• To design specialized methods for the low-dimensional case (up to 5 dimensions) that exploit multidimensional dichotomy and inexact gradient oracles.
• To provide theoretical complexity bounds that match or approach the known lower bounds for this class of problems.
• To unify and extend existing methods by combining accelerated gradient schemes with inexact subproblem solves for the high-dimensional variable and exact or efficient solvers for the low-dimensional one.

Proposed method

• The method decomposes the original saddle point problem into a sequence of subproblems by fixing one variable and solving for the other, exploiting the small dimension of one variable group.
• For the high-dimensional variable, the algorithm uses accelerated gradient methods with inexact gradient information, where the gradient is approximated via solving a subproblem over the low-dimensional variable.
• For the low-dimensional variable, the algorithm applies the ellipsoid method or Polyak's method to solve the subproblems exactly or with high accuracy.
• In the special case of very small dimension (≤5), a new multidimensional dichotomy method is proposed, which allows for efficient minimization over a hypercube using inexact gradient oracles.
• The algorithm ensures convergence by recursively reducing the search space and maintaining bounds on the error in gradient approximation using Lipschitz continuity and strong convexity parameters.
• Theoretical complexity is derived by analyzing the required accuracy of subproblem solutions and the number of iterations needed to achieve the target accuracy $\varepsilon$.

Experimental results

Research questions

• RQ1 Can we achieve near-optimal complexity for strongly convex-concave saddle problems when one variable group has small dimension and the other has large dimension?
• RQ2 How can we efficiently compute inexact gradient information for the high-dimensional variable using subproblems over the low-dimensional variable?
• RQ3 What is the optimal trade-off between solving the low-dimensional subproblems accurately and the overall convergence rate of the main algorithm?
• RQ4 Can a new multidimensional dichotomy method improve convergence for very small-dimensional cases (≤5) with inexact gradient oracles?
• RQ5 How do the theoretical complexity bounds of the proposed method compare to the known lower bounds for this class of problems?

Key findings

• The proposed algorithm achieves a complexity bound of $\widetilde{O}\left(\sqrt{\frac{L_{xx}}{\mu_x} + \frac{L_{xy}^2}{\mu_x\mu_y} + \frac{L_{yy}}{\mu_y}}\right)$ for the general case of small-dimensional variable groups.
• For the special case of dimension ≤5, the algorithm achieves a complexity bound of $\widetilde{O}\left(\sqrt{\frac{L_{xx}}{\mu_x} + \frac{L_{yy}}{\mu_y}}\right)$ using a new multidimensional dichotomy method with inexact gradients.
• The algorithm ensures that the required accuracy of the gradient approximation in the high-dimensional subproblems is bounded by $\Delta \leq \frac{1}{4} \cdot \frac{L_f}{M_f + 2L_f R} \cdot \varepsilon$, which determines the number of iterations in the subproblem solve.
• The number of iterations required for the subproblem solve is bounded by $N = \left\lceil \log_2\left(\frac{4R(M_f + 2L_f R)}{L_f \varepsilon}\right) \right\rceil$, which ensures the overall convergence to $\varepsilon$-accuracy.
• The method achieves near-optimal complexity by combining inexact gradient information with efficient solvers for low-dimensional problems, matching the known lower bound up to logarithmic factors.
• Theoretical analysis confirms that the method is robust to inexactness in gradient computation, as long as the error is controlled via the derived bounds on $\Delta$.