[Paper Review] Kantorovich Strikes Back! Wasserstein GANs are not Optimal Transport?

Alexander Korotin, A. K. Kolesov+1 more|arXiv (Cornell University)|Jun 15, 2022

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper evaluates the performance of various Wasserstein GAN dual solvers in estimating optimal transport (OT) metrics, particularly the 1-Wasserstein distance (W₁) and its gradient. Using a novel benchmark with known ground-truth Kantorovich potentials, it shows that while most solvers recover OT gradients with positive cosine similarity, they fail to accurately estimate W₁ in high dimensions—especially DOT, which overestimates W₁ exponentially with dimensionality, undermining their use as precise OT estimators despite utility in GAN training.

ABSTRACT

Wasserstein Generative Adversarial Networks (WGANs) are the popular generative models built on the theory of Optimal Transport (OT) and the Kantorovich duality. Despite the success of WGANs, it is still unclear how well the underlying OT dual solvers approximate the OT cost (Wasserstein-1 distance, $\mathbb{W}_{1}$) and the OT gradient needed to update the generator. In this paper, we address these questions. We construct 1-Lipschitz functions and use them to build ray monotone transport plans. This strategy yields pairs of continuous benchmark distributions with the analytically known OT plan, OT cost and OT gradient in high-dimensional spaces such as spaces of images. We thoroughly evaluate popular WGAN dual form solvers (gradient penalty, spectral normalization, entropic regularization, etc.) using these benchmark pairs. Even though these solvers perform well in WGANs, none of them faithfully compute $\mathbb{W}_{1}$ in high dimensions. Nevertheless, many provide a meaningful approximation of the OT gradient. These observations suggest that these solvers should not be treated as good estimators of $\mathbb{W}_{1}$, but to some extent they indeed can be used in variational problems requiring the minimization of $\mathbb{W}_{1}$.

Motivation & Objective

• To develop a benchmark for evaluating dual solvers of optimal transport (OT) in high-dimensional and image data settings.
• To assess the accuracy of WGAN dual solvers in estimating the 1-Wasserstein distance (W₁) and its gradient.
• To identify which solvers provide reliable OT cost and gradient estimates despite known limitations in high-dimensional spaces.
• To evaluate the practical utility of these solvers in generative modeling, particularly in Wasserstein GANs.
• To establish a reproducible, transparent benchmark for future OT and WGAN research using known ground-truth OT potentials.

Proposed method

• Constructs high-dimensional and image-based benchmark pairs of continuous distributions with known ground-truth optimal Kantorovich potentials using MinFunnels.
• Employs the Kantorovich duality formulation to generate pairs (P, Q) supported on [-1.1, 1.1]^D with known W₁ and ∇f*.
• Evaluates 12 WGAN dual solvers (e.g., GP, LP, SN, SO, LS, MM, DOT) on their ability to estimate W₁ and the gradient ∇f*.
• Measures performance using three metrics: cosine similarity (cos) between estimated and true gradients, L² norm of gradient error, and W₁ estimation accuracy.
• Uses DCGAN and UNet architectures for potentials and movers, with spectral normalization, GroupSort, and orthogonality constraints where applicable.
• Employs Adam optimizer with fixed hyperparameters and trains until test cosine similarity converges, ensuring stable evaluation.

Experimental results

Research questions

• RQ1 How accurately do WGAN dual solvers estimate the 1-Wasserstein distance (W₁) in high-dimensional and image-based distributions?
• RQ2 To what extent do these solvers recover the true optimal transport gradient, as measured by cosine similarity with ground truth?
• RQ3 Why do some solvers like DOT fail in high dimensions despite working in low dimensions?
• RQ4 Can solvers with poor W₁ estimation still be effective in training Wasserstein GANs?
• RQ5 How do architectural choices (e.g., DCGAN vs. UNet, normalization techniques) impact the accuracy of OT estimation?

Key findings

• Empirical DOT overestimates W₁ in high dimensions due to exponential sample complexity (in D), leading to poor W₁ estimation despite good gradient alignment.
• DOT shows cos ≈ 0 in high dimensions, indicating its gradient is nearly orthogonal to the true gradient, despite being used in WGANs.
• MM:R achieves the best L² gradient error (cos > 0.9) and precise W₁ estimation in images, outperforming others in both metrics.
• MM:Bv2 estimates W₁ reasonably but has cos ≈ 0 and high L² error, indicating poor gradient recovery despite decent W₁ estimation.
• GP, LP, and MM:R achieve cos > 0.9 in images, indicating strong gradient alignment, though only MM:R provides accurate L² gradient estimates.
• WC and SN show moderate cos > 0 but significantly lower than top performers, indicating suboptimal gradient recovery in high dimensions.