[Paper Review] Sample Complexity of Sinkhorn divergences

Aude Genevay, Lénaïc Chizat+3 more|arXiv (Cornell University)|Oct 5, 2018

Probabilistic and Robust Engineering DesignDecision Sciences142 citations

TL;DR

The paper derives a new sample complexity bound for Sinkhorn divergences by reformulating them in an RKHS; it shows a 1/√n rate with a regularization-dependent constant, bridging OT and MMD.

ABSTRACT

Optimal transport (OT) and maximum mean discrepancies (MMD) are now routinely used in machine learning to compare probability measures. We focus in this paper on \emph{Sinkhorn divergences} (SDs), a regularized variant of OT distances which can interpolate, depending on the regularization strength $\varepsilon$, between OT ($\varepsilon=0$) and MMD ($\varepsilon=\infty$). Although the tradeoff induced by that regularization is now well understood computationally (OT, SDs and MMD require respectively $O(n^3\log n)$, $O(n^2)$ and $n^2$ operations given a sample size $n$), much less is known in terms of their \emph{sample complexity}, namely the gap between these quantities, when evaluated using finite samples \emph{vs.} their respective densities. Indeed, while the sample complexity of OT and MMD stand at two extremes, $1/n^{1/d}$ for OT in dimension $d$ and $1/\sqrt{n}$ for MMD, that for SDs has only been studied empirically. In this paper, we \emph{(i)} derive a bound on the approximation error made with SDs when approximating OT as a function of the regularizer $\varepsilon$, \emph{(ii)} prove that the optimizers of regularized OT are bounded in a Sobolev (RKHS) ball independent of the two measures and \emph{(iii)} provide the first sample complexity bound for SDs, obtained,by reformulating SDs as a maximization problem in a RKHS. We thus obtain a scaling in $1/\sqrt{n}$ (as in MMD), with a constant that depends however on $\varepsilon$, making the bridge between OT and MMD complete.

Motivation & Objective

• Motivate the study of sample complexity for regularized OT in high dimensions.
• Derive a bound on the approximation error between regularized OT and standard OT as a function of the entropic regularization parameter ε.
• Show that Sinkhorn optimizers lie in a Sobolev (RKHS) ball independent of the input measures.
• Reformulate Sinkhorn divergences as an RKHS expectation maximization problem.
• Provide the first explicit sample complexity bound for Sinkhorn divergences and relate it to MMD and OT.

Proposed method

• Establish a bound between the entropic-regularized OT Wε and the true OT W, as a function of ε.
• Prove that Sinkhorn potentials are bounded in a Sobolev (RKHS) space, independent of the marginals.
• Reformulate the SD as an RKHS-based max of an expectation to enable kernel-SGD approaches.
• Apply RKHS-based PAC-learning (Bartlett–Mendelson framework) to obtain a 1/√n convergence rate for the empirical SD.
• Derive ε-dependent constants and asymptotics for the convergence and provide corollaries for concentration.

Experimental results

Research questions

• RQ1 How does the entropic regularization parameter ε affect the approximation error Wε - W between regularized OT and standard OT?
• RQ2 Can Sinkhorn potentials be bounded in a Sobolev/RKHS ball independent of marginals, enabling RKHS-based optimization methods?
• RQ3 What is the sample complexity of Sinkhorn divergences when estimated from finite samples, and how does it scale with n and ε?
• RQ4 How does SD interpolate between OT (ε→0) and MMD (ε→∞) in terms of statistical efficiency?
• RQ5 What are the practical implications for kernel-SGD and RKHS-based optimization in computing SDs?

Key findings

• Wε(α,β) − W(α,β) ≤ 2ε d log(e^2 L D /(√d ε)) and asymptotically ~ 2ε d log(1/ε) as ε→0.
• Sinkhorn potentials (u,v) are uniformly bounded in the Sobolev space Hs(R^d) with norms O(1+1/ε^{s−1}).
• The optimizers of the regularized OT problem lie in an RKHS ball independent of the measures, enabling a kernel-based optimization approach.
• Empirical Sinkhorn divergence converges to the population value at rate O(1/√n) with a constant that scales like exp(κ/ε)/ε^{⌊d/2⌋} for small ε and becomes ε-independent for large ε.
• A PAC/RKHS-based analysis yields a bound on the empirical SD error: E|Wε(α,β) − Wε(α̂n,β̂n)| = O((e^{κ/ε}/√n)(1+1/ε^{⎣d/2⎦})).
• Corollaries include concentration bounds showing high-probability control of the empirical error.