[Paper Review] From Smooth Wasserstein Distance to Dual Sobolev Norm: Empirical Approximation and Statistical Applications

Sloan Nietert, Ziv Goldfeld+1 more|arXiv (Cornell University)|Jan 11, 2021

Topological and Geometric Data Analysis70 references4 citations

TL;DR

This paper introduces the Gaussian-smoothed $p$-Wasserstein distance $\mathsf{W}_p^{(\sigma)}$ to overcome the curse of dimensionality in statistical estimation. By showing it is controlled by a dual Sobolev norm $\mathsf{d}_p^{(\sigma)}$, the authors establish a parametric $n^{-1/2}$ convergence rate in high dimensions, contrasting the $n^{-1/d}$ rate of unsmoothed Wasserstein distances, and derive asymptotic distributions for two-sample testing and minimum distance estimation.

ABSTRACT

Statistical distances, i.e., discrepancy measures between probability distributions, are ubiquitous in probability theory, statistics and machine learning. To combat the curse of dimensionality when estimating these distances from data, recent work has proposed smoothing out local irregularities in the measured distributions via convolution with a Gaussian kernel. Motivated by the scalability of the smooth framework to high dimensions, we conduct an in-depth study of the structural and statistical behavior of the Gaussian-smoothed $p$-Wasserstein distance $\mathsf{W}_p^{(\sigma)}$, for arbitrary $p\geq 1$. We start by showing that $\mathsf{W}_p^{(\sigma)}$ admits a metric structure that is topologically equivalent to classic $\mathsf{W}_p$ and is stable with respect to perturbations in $\sigma$. Moving to statistical questions, we explore the asymptotic properties of $\mathsf{W}_p^{(\sigma)}(\hat{\mu}_n,\mu)$, where $\hat{\mu}_n$ is the empirical distribution of $n$ i.i.d. samples from $\mu$. To that end, we prove that $\mathsf{W}_p^{(\sigma)}$ is controlled by a $p$th order smooth dual Sobolev norm $\mathsf{d}_p^{(\sigma)}$. Since $\mathsf{d}_p^{(\sigma)}(\hat{\mu}_n,\mu)$ coincides with the supremum of an empirical process indexed by Gaussian-smoothed Sobolev functions, it lends itself well to analysis via empirical process theory. We derive the limit distribution of $\sqrt{n}\mathsf{d}_p^{(\sigma)}(\hat{\mu}_n,\mu)$ in all dimensions $d$, when $\mu$ is sub-Gaussian. Through the aforementioned bound, this implies a parametric empirical convergence rate of $n^{-1/2}$ for $\mathsf{W}_p^{(\sigma)}$, contrasting the $n^{-1/d}$ rate for unsmoothed $\mathsf{W}_p$ when $d \geq 3$. As applications, we provide asymptotic guarantees for two-sample testing and minimum distance estimation. When $p=2$, we further show that $\mathsf{d}_2^{(\sigma)}$ can be expressed as a maximum mean discrepancy.

Motivation & Objective

• To address the curse of dimensionality in estimating statistical distances between probability distributions.
• To develop a scalable, stable framework for $p$-Wasserstein distances in high-dimensional settings.
• To establish asymptotic theory for empirical estimation of the smoothed Wasserstein distance using empirical process methods.
• To provide statistical guarantees for two-sample testing and minimum distance estimation under the smoothed distance.
• To characterize the relationship between the smoothed Wasserstein distance and dual Sobolev norms.

Proposed method

• Introduce the Gaussian-smoothed $p$-Wasserstein distance $\mathsf{W}_p^{(\sigma)}$ by convolving distributions with a Gaussian kernel to reduce local irregularities.
• Prove that $\mathsf{W}_p^{(\sigma)}$ is topologically equivalent to the standard $\mathsf{W}_p$ and stable under perturbations in $\sigma$.
• Establish a control of $\mathsf{W}_p^{(\sigma)}$ by the $p$th-order dual Sobolev norm $\mathsf{d}_p^{(\sigma)}$, which measures the supremum of an empirical process over Gaussian-smoothed Sobolev functions.
• Use empirical process theory to derive the asymptotic distribution of $\sqrt{n}\,\mathsf{d}_p^{(\sigma)}(\hat{\mu}_n, \mu)$ under sub-Gaussianity of $\mu$.
• Show that when $p=2$, $\mathsf{d}_2^{(\sigma)}$ coincides with the maximum mean discrepancy (MMD) under Gaussian smoothing.
• Leverage the dual Sobolev norm to derive parametric convergence rates and statistical inference guarantees.

Experimental results

Research questions

• RQ1 Can the Gaussian-smoothed $p$-Wasserstein distance $\mathsf{W}_p^{(\sigma)}$ achieve a faster convergence rate than the unsmoothed $\mathsf{W}_p$ in high dimensions?
• RQ2 How does the dual Sobolev norm $\mathsf{d}_p^{(\sigma)}$ relate to the smoothed Wasserstein distance $\mathsf{W}_p^{(\sigma)}$?
• RQ3 What is the asymptotic distribution of $\sqrt{n}\,\mathsf{d}_p^{(\sigma)}(\hat{\mu}_n, \mu)$ for sub-Gaussian $\mu$?
• RQ4 Can the smoothed Wasserstein distance support valid two-sample testing and minimum distance estimation with asymptotic guarantees?
• RQ5 Is there a connection between $\mathsf{d}_2^{(\sigma)}$ and the maximum mean discrepancy (MMD)?

Key findings

• The Gaussian-smoothed $p$-Wasserstein distance $\mathsf{W}_p^{(\sigma)}$ is topologically equivalent to the standard $\mathsf{W}_p$ and stable under small changes in $\sigma$.
• The distance $\mathsf{W}_p^{(\sigma)}$ is controlled by the dual Sobolev norm $\mathsf{d}_p^{(\sigma)}$, which allows for empirical process analysis.
• For sub-Gaussian $\mu$, the asymptotic distribution of $\sqrt{n}\,\mathsf{d}_p^{(\sigma)}(\hat{\mu}_n, \mu)$ is derived in all dimensions $d$.
• The empirical convergence rate of $\mathsf{W}_p^{(\sigma)}$ is $n^{-1/2}$, which is parametric and independent of dimension, contrasting the $n^{-1/d}$ rate of unsmoothed $\mathsf{W}_p$ when $d \geq 3$.
• When $p=2$, the dual Sobolev norm $\mathsf{d}_2^{(\sigma)}$ is equivalent to the maximum mean discrepancy under Gaussian smoothing.
• Asymptotic guarantees are established for two-sample testing and minimum distance estimation using the smoothed distance framework.