[Paper Review] On the Efficiency of the Sinkhorn and Greenkhorn Algorithms and Their Acceleration for Optimal Transport

Tianyi Lin, Nhat Ho+1 more|arXiv (Cornell University)|Jun 1, 2019

Stochastic Gradient Optimization Techniques49 references14 citations

TL;DR

This paper presents improved complexity bounds for regularized optimal transport algorithms, showing that the Greenkhorn algorithm achieves $×sim\mathcal{O}(n^2\varepsilon^{-2})$ complexity, matching Sinkhorn’s best-known bound and explaining its empirical superiority. It further introduces accelerated variants, including an adaptive primal-dual mirror descent algorithm with $×sim\mathcal{O}(n^{7/3}\varepsilon^{-1})$ complexity, outperforming prior methods in terms of $ \varepsilon$-dependence.

ABSTRACT

We present new complexity results for several algorithms that approximately solve the regularized optimal transport (OT) problem between two discrete probability measures with at most $n$ atoms. First, we show that a greedy variant of the classical Sinkhorn algorithm, known as the extit{Greenkhorn} algorithm, achieves the complexity bound of $\widetilde{\mathcal{O}}(n^2\varepsilon^{-2})$, which improves the best known bound $\widetilde{\mathcal{O}}(n^2\varepsilon^{-3})$. Notably, this matches the best known complexity bound of the Sinkhorn algorithm and explains the superior performance of the Greenkhorn algorithm in practice. Furthermore, we generalize an adaptive primal-dual accelerated gradient descent (APDAGD) algorithm with mirror mapping $\phi$ and show that the resulting extit{adaptive primal-dual accelerated mirror descent} (APDAMD) algorithm achieves the complexity bound of $\widetilde{\mathcal{O}}(n^2\sqrt{\delta}\varepsilon^{-1})$ where $\delta>0$ depends on $\phi$. We point out that an existing complexity bound for the APDAGD algorithm is not valid in general using a simple counterexample and then establish the complexity bound of $\widetilde{\mathcal{O}}(n^{5/2}\varepsilon^{-1})$ by exploiting the connection between the APDAMD and APDAGD algorithms. Moreover, we introduce accelerated Sinkhorn and Greenkhorn algorithms that achieve the complexity bound of $\widetilde{\mathcal{O}}(n^{7/3}\varepsilon^{-1})$, which improves on the complexity bounds $\widetilde{\mathcal{O}}(n^2\varepsilon^{-2})$ of Sinkhorn and Greenkhorn algorithms in terms of $\varepsilon$. Experimental results on synthetic and real datasets demonstrate the favorable performance of new algorithms in practice.

Motivation & Objective

• To establish tighter complexity bounds for the Greenkhorn algorithm, which empirically outperforms standard Sinkhorn in optimal transport.
• To correct and generalize an existing adaptive primal-dual accelerated gradient descent (APDAGD) algorithm with mirror mapping.
• To develop and analyze accelerated versions of Sinkhorn and Greenkhorn that improve convergence rates in terms of the accuracy parameter $ \varepsilon$.
• To provide a rigorous complexity analysis of the APDAMD algorithm and clarify the validity of prior bounds for APDAGD using a counterexample.
• To validate the theoretical improvements through experiments on synthetic and real-world datasets.

Proposed method

• Propose a greedy variant of Sinkhorn, named Greenkhorn, which updates only the most imbalanced dual variables at each iteration, reducing computational waste.
• Introduce the adaptive primal-dual accelerated mirror descent (APDAMD) algorithm using a general mirror mapping $ \phi$, enabling flexible and adaptive updates.
• Correct a previously claimed complexity bound for APDAGD by providing a counterexample, demonstrating its invalidity in general settings.
• Establish a new complexity bound of $×sim\mathcal{O}(n^{5/2}\varepsilon^{-1})$ for APDAMD by leveraging its connection to APDAGD.
• Design accelerated Sinkhorn and Greenkhorn algorithms using Nesterov-style acceleration, achieving $×sim\mathcal{O}(n^{7/3}\varepsilon^{-1})$ complexity.
• Use theoretical analysis grounded in convex optimization and duality to derive convergence rates under regularized optimal transport settings.

Experimental results

Research questions

• RQ1 What is the true complexity of the Greenkhorn algorithm, and does it match or exceed that of the standard Sinkhorn algorithm?
• RQ2 Can the APDAGD algorithm’s claimed complexity bound be universally applied, or are there conditions under which it fails?
• RQ3 How can mirror descent be adapted to achieve accelerated convergence in regularized optimal transport problems?
• RQ4 Can the theoretical gains of acceleration be extended to the Greenkhorn framework, and what complexity bounds result?
• RQ5 Do the proposed accelerated algorithms outperform existing methods in practice, especially in terms of $ \varepsilon$-dependence?

Key findings

• The Greenkhorn algorithm achieves a complexity bound of $×sim\mathcal{O}(n^2\varepsilon^{-2})$, matching the best-known bound for the standard Sinkhorn algorithm.
• The APDAMD algorithm achieves a complexity bound of $×sim\mathcal{O}(n^2\sqrt{\delta}\varepsilon^{-1})$, where $ \delta>0$ depends on the mirror mapping $ \phi$, generalizing prior results.
• A counterexample is provided to invalidate a previously claimed complexity bound for the APDAGD algorithm, showing it does not hold in general.
• By connecting APDAMD to APDAGD, the paper establishes a new complexity bound of $×sim\mathcal{O}(n^{5/2}\varepsilon^{-1})$ for the APDAMD framework.
• The proposed accelerated Sinkhorn and Greenkhorn algorithms achieve a complexity of $×sim\mathcal{O}(n^{7/3}\varepsilon^{-1})$, improving over the $×sim\mathcal{O}(n^2\varepsilon^{-2})$ bound of standard variants in terms of $ \varepsilon$.
• Experimental results on synthetic and real datasets confirm that the new accelerated algorithms achieve faster convergence and better performance in practice.