[Paper Review] On accelerated alternating minimization

Sergey Guminov, Pavel Dvurechensky+1 more|arXiv (Cornell University)|Jan 1, 2020

Matrix Theory and Algorithms10 references15 citations

TL;DR

This paper proposes an accelerated alternating minimization algorithm with a $1/k^2$ convergence rate for entropy-regularized optimal transport problems, significantly improving upon the $1/k$ rate of standard alternating minimization and Sinkhorn’s algorithm. The method is primal-dual, adaptive to local smoothness, and outperforms gradient-based and Sinkhorn methods in practice without requiring step-size tuning.

ABSTRACT

Alternating minimization (AM) optimization algorithms have been known for a long time and are of importance in machine learning problems, among which we are mostly motivated by approximating optimal transport distances. AM algorithms assume that the decision variable is divided into several blocks and minimization in each block can be done explicitly or cheaply with high accuracy. The ubiquitous Sinkhorn's algorithm can be seen as an alternating minimization algorithm for the dual to the entropy-regularized optimal transport problem. We introduce an accelerated alternating minimization method with a $1/k^2$ convergence rate, where $k$ is the iteration counter. This improves over known bound $1/k$ for general AM methods and for the Sinkhorn's algorithm. Moreover, our algorithm converges faster than gradient-type methods in practice as it is free of the choice of the step-size and is adaptive to the local smoothness of the problem. We show that the proposed method is primal-dual, meaning that if we apply it to a dual problem, we can reconstruct the solution of the primal problem with the same convergence rate. We apply our method to the entropy regularized optimal transport problem and show experimentally, that it outperforms Sinkhorn's algorithm.

Motivation & Objective

• To address the slow convergence of standard alternating minimization and Sinkhorn’s algorithm in entropy-regularized optimal transport problems.
• To develop an accelerated variant of alternating minimization that achieves a faster $1/k^2$ convergence rate compared to the standard $1/k$ rate.
• To design a method that is adaptive to local problem smoothness and avoids the need for step-size selection.
• To ensure the algorithm maintains primal-dual structure, enabling recovery of the primal solution with the same convergence rate.

Proposed method

• The proposed method applies Nesterov-style acceleration to alternating minimization, incorporating momentum terms that depend on the iteration counter $k$.
• It alternates between minimizing the dual objective with respect to each block of variables, using closed-form updates when possible.
• The acceleration mechanism ensures a global $1/k^2$ convergence rate for the dual objective, improving over the standard $1/k$ rate.
• The algorithm is primal-dual: solving the dual problem yields the primal solution with the same $1/k^2$ convergence rate.
• The method is adaptive to local smoothness, eliminating the need for manual step-size selection common in gradient-type methods.
• The approach is applied to the entropy-regularized optimal transport problem, where it leverages the structure of the dual to enable efficient updates.

Experimental results

Research questions

• RQ1 Can alternating minimization be accelerated to achieve a $1/k^2$ convergence rate in entropy-regularized optimal transport problems?
• RQ2 Does the proposed acceleration method outperform Sinkhorn’s algorithm and gradient-based methods in practice?
• RQ3 Can the algorithm maintain a primal-dual structure while achieving faster convergence?
• RQ4 Is the method adaptive to local problem smoothness, eliminating the need for step-size tuning?
• RQ5 Can the accelerated alternating minimization be efficiently implemented for large-scale optimal transport problems?

Key findings

• The proposed accelerated alternating minimization achieves a $1/k^2$ convergence rate for the dual objective, improving upon the standard $1/k$ rate of general alternating minimization and Sinkhorn’s algorithm.
• The method outperforms Sinkhorn’s algorithm in practice, demonstrating faster convergence on benchmark optimal transport problems.
• The algorithm is adaptive to local smoothness, eliminating the need for step-size selection and improving robustness.
• The primal-dual structure allows the recovery of the primal solution with the same $1/k^2$ convergence rate as the dual.
• Empirical results confirm that the method converges faster than gradient-type methods due to its adaptive nature and lack of step-size dependency.