[Paper Review] Computational Optimal Transport: Complexity by Accelerated Gradient Descent Is Better Than by Sinkhorn's Algorithm

Pavel Dvurechensky, Alexander Gasnikov+1 more|arXiv (Cornell University)|Feb 12, 2018

Stochastic Gradient Optimization Techniques2 references68 citations

TL;DR

The paper analyzes two algorithms for approximating OT distances between discrete distributions: Sinkhorn with entropy regularization and a novel Adaptive Primal-Dual Accelerated Gradient Descent (APDAGD); APDAGD achieves better ε-dependence and supports general regularizers beyond entropic regularization.

ABSTRACT

We analyze two algorithms for approximating the general optimal transport (OT) distance between two discrete distributions of size $n$, up to accuracy $\varepsilon$. For the first algorithm, which is based on the celebrated Sinkhorn's algorithm, we prove the complexity bound $\widetilde{O}\left({n^2/\varepsilon^2} ight)$ arithmetic operations. For the second one, which is based on our novel Adaptive Primal-Dual Accelerated Gradient Descent (APDAGD) algorithm, we prove the complexity bound $\widetilde{O}\left(\min\left\{n^{9/4}/\varepsilon, n^{2}/\varepsilon^2 ight\} ight)$ arithmetic operations. Both bounds have better dependence on $\varepsilon$ than the state-of-the-art result given by $\widetilde{O}\left({n^2/\varepsilon^3} ight)$. Our second algorithm not only has better dependence on $\varepsilon$ in the complexity bound, but also is not specific to entropic regularization and can solve the OT problem with different regularizers.

Motivation & Objective

• Motivate the need for efficient computation of OT distances between equal-sized discrete distributions.
• Provide improved complexity bounds for entropy-regularized OT using Sinkhorn’s algorithm.
• Introduce APDAGD as a flexible, accelerated method with line-search for general regularizers.
• Derive theoretical guarantees and ε-dependent complexity for solving OT with APDAGD.
• Demonstrate practical performance through numerical experiments on image-like data.

Proposed method

• Revisit Sinkhorn’s algorithm as dual updates solving a regularized OT dual problem, and derive improved iteration bounds.
• Provide an Adaptive Primal-Dual Accelerated Gradient Descent (APDAGD) with line-search and primal-dual updates for general strongly convex regularizers.
• Apply APDAGD to entropy-regularized OT and derive convergence and complexity guarantees.
• Develop Algorithm 4 that combines APDAGD with projection to the transportation polytope U(r,c) for OT distance approximation.
• Show that with entropy regularization, the method attains an ε-approximation with favorable ε- and ∥C∥∞-dependence.
• Discuss parallelizability and practical considerations for kernel-expensive OT computations.

Experimental results

Research questions

• RQ1 Can entropy-regularized OT be approximated more efficiently than existing bounds when using Sinkhorn’s algorithm?
• RQ2 Does an adaptive primal-dual accelerated gradient approach provide faster ε-accurate OT approximations with general regularizers?
• RQ3 What are the ε-dependence and norm-constant dependencies of the proposed APDAGD method for OT?
• RQ4 Can the framework handle non-entropic regularizers beyond entropy while preserving convergence guarantees?
• RQ5 How do the practical performances of Sinkhorn and APDAGD compare on discretized distributions with varying n and regularization?

Key findings

• Sinkhorn's algorithm can approximate OT within ε in O(n^2 ∥C∥^2_∞ ln n / ε^2) arithmetic operations.
• APDAGD achieves an ε-approximation with O(min(n^(9/4)/(√ε) ∥C∥∞ ln n / ε, n^2 ∥C∥∞ ln n / ε^2)) arithmetic operations.
• The APDAGD method is not restricted to entropic regularization and works with general strongly convex regularizers.
• APDAGD includes line-search and an online stopping criterion based on duality gap and constraint infeasibility.
• The framework supports parallelization and is practical when the OT kernel exp(−C/γ) is efficiently applicable.
• Empirical experiments on MNIST-derived data illustrate practical performance differences between Sinkhorn and APDAGD.