[Paper Review] Near-linear time approximation algorithms for optimal transport via Sinkhorn iteration

Jason M. Altschuler, Jonathan Weed+1 more|arXiv (Cornell University)|May 26, 2017

Advanced Bandit Algorithms Research35 references211 citations

TL;DR

The paper proves near-linear time algorithms for approximating optimal transport distances using Sinkhorn iteration, and introduces a greedy variant Greenkhorn with the same guarantees and practical advantages.

ABSTRACT

Computing optimal transport distances such as the earth mover's distance is a fundamental problem in machine learning, statistics, and computer vision. Despite the recent introduction of several algorithms with good empirical performance, it is unknown whether general optimal transport distances can be approximated in near-linear time. This paper demonstrates that this ambitious goal is in fact achieved by Cuturi's Sinkhorn Distances. This result relies on a new analysis of Sinkhorn iteration, which also directly suggests a new greedy coordinate descent algorithm, Greenkhorn, with the same theoretical guarantees. Numerical simulations illustrate that Greenkhorn significantly outperforms the classical Sinkhorn algorithm in practice.

Motivation & Objective

• Motivate fast computation of optimal transport distances on discrete measures.
• Provide near-linear time approximation guarantees for OT via entropic regularization.
• Introduce practical, easy-to-implement algorithms with theoretical guarantees.
• Offer a simple rounding procedure to obtain feasible OT solutions from approximations.

Proposed method

• Formulate OT as a linear program with marginals r and c.
• Use entropic regularization to define the Sinkhorn projection P_eta and its explicit form P_eta = X A Y with A = exp(-eta C).
• Develop an approximate Sinkhorn projection Proj(A, U_{r,c}, epsilon') that yields a B close to the transport polytope in marginal distance.
• Provide a rounding routine round(F, U_{r,c}) to produce a feasible P in U_{r,c} with controlled error.
• Analyze two projection methods: Sinkhorn (iterative row/column scaling) and Greenkhorn (greedy coordinate updates) with near-linear time guarantees.
• Prove an overall runtime: O(n^2) plus S, where S = O(n^2 L^3 (log n) epsilon^{-3}) when ||C||_infty <= L.

Experimental results

Research questions

• RQ1 Can OT distances between discrete measures be approximated in near-linear time in the input size n^2?
• RQ2 Does entropic regularization enable rigorous near-linear time approximation guarantees for general non-negative cost matrices C?
• RQ3 Can a practical, easy-to-implement variant of Sinkhorn (Greenkhorn) achieve the same guarantees with better empirical performance?
• RQ4 How can a rounding step convert an approximate Sinkhorn projection into a feasible transport plan with provable error bounds?

Key findings

• The authors show that Sinkhorn distances with entropic regularization yield near-linear time approximations for OT distances between discrete measures.
• They provide a precise runtime bound: O(n^2 + S), with S = O(n^2 L^3 log n epsilon^{-3}) when ||C||_infty <= L.
• A greedy coordinate descent variant, Greenkhorn, achieves the same theoretical guarantees as Sinkhorn but with improved practical performance.
• A simple rounding procedure yields a feasible transport plan in U_{r,c} with additive error epsilon in the OT objective.
• The paper supplies guidance on parameter tuning for the Sinkhorn approach and demonstrates empirical superiority of Greenkhorn over standard Sinkhorn on image data (MNIST) and synthetic experiments.
• Empirical results indicate Greenkhorn significantly outperforms Sinkhorn in both short and long runs.