[Paper Review] A gradual, semi-discrete approach to generative network training via explicit Wasserstein minimization

Yucheng Chen, Matus Telgarsky+4 more|arXiv (Cornell University)|Jun 8, 2019

Generative Adversarial Networks and Image Synthesis34 references4 citations

TL;DR

This paper proposes a novel, non-adversarial generative modeling approach that alternately performs optimal transport mapping and regression to explicitly minimize the Wasserstein distance between a generator's output and a target distribution. By leveraging a semi-discrete setting with continuous generator outputs, it achieves provable minimization of both empirical and population Wasserstein distances, demonstrating state-of-the-art performance on MNIST and Thin-8 with improved generalization and mode coverage.

ABSTRACT

This paper provides a simple procedure to fit generative networks to target distributions, with the goal of a small Wasserstein distance (or other optimal transport costs). The approach is based on two principles: (a) if the source randomness of the network is a continuous distribution (the "semi-discrete" setting), then the Wasserstein distance is realized by a deterministic optimal transport mapping; (b) given an optimal transport mapping between a generator network and a target distribution, the Wasserstein distance may be decreased via a regression between the generated data and the mapped target points. The procedure here therefore alternates these two steps, forming an optimal transport and regressing against it, gradually adjusting the generator network towards the target distribution. Mathematically, this approach is shown to minimize the Wasserstein distance to both the empirical target distribution, and also its underlying population counterpart. Empirically, good performance is demonstrated on the training and testing sets of the MNIST and Thin-8 data. The paper closes with a discussion of the unsuitability of the Wasserstein distance for certain tasks, as has been identified in prior work [Arora et al., 2017, Huang et al., 2017].

Motivation & Objective

• To develop a non-adversarial, alternating procedure that explicitly minimizes the Wasserstein distance between a generative network's output and a target data distribution.
• To address the limitation of batch-to-batch optimal transport approximations, which fail to minimize the true Wasserstein distance due to sampling bias.
• To ensure generalization to the underlying data distribution, not just the empirical training set, by proving bounds that are polynomial in dimension rather than exponential.
• To demonstrate improved performance in mode coverage and distributional fidelity compared to GANs and VAEs, particularly on challenging datasets like Thin-8.
• To explore the trade-off between low Wasserstein distance and visual sharpness, and propose remedies such as perceptual loss or adversarial regularization.

Proposed method

• The method alternates between two steps: Optimal Transport Solver (OTS), which computes a deterministic optimal transport map between the current generator’s output distribution and the target distribution in a semi-discrete setting.
• The FIT step performs regression to update the generator network by shifting its generated samples toward the target points as defined by the OT map, using standard regression techniques.
• The semi-discrete formulation ensures that the optimal transport is deterministic and exact, avoiding the bias inherent in batch-to-batch transport approximations.
• The procedure is gradual, slowly deforming the generator’s output distribution toward the target, which empirically leads to better generalization and smoother manifolds.
• Theoretical guarantees are established using the triangle inequality and concentration bounds, showing that the method minimizes the Wasserstein distance to both the empirical dataset and the underlying population distribution.
• The approach is applied to MNIST, CIFAR10, and the challenging Thin-8 dataset, with comparisons to GANs and VAEs using Wasserstein-1 distance as a metric.

Experimental results

Research questions

• RQ1 Can a non-adversarial, alternating procedure explicitly minimize the Wasserstein distance between a generative model’s output and a target distribution?
• RQ2 Does the semi-discrete formulation of optimal transport enable exact and deterministic mapping, avoiding the bias of batch-to-batch approximations?
• RQ3 Can the gradual, iterative refinement process lead to better generalization to the underlying data distribution rather than just the training set?
• RQ4 How does minimizing the Wasserstein distance affect mode coverage and visual quality compared to GAN-based methods?
• RQ5 What are the limitations of Wasserstein minimization in terms of image sharpness, and how can they be mitigated?

Key findings

• The method achieves the lowest Wasserstein-1 distance on CIFAR10 (655), outperforming WGAN-GP (849) and VAE (745), indicating superior distributional alignment.
• On MNIST and Thin-8, the method generates qualitatively better digits than baselines, with consistent performance on both training and test sets.
• The approach prevents mode collapse naturally due to the high transport cost of omitting modes, leading to better coverage of data distribution modes.
• Despite lower blurriness in generated samples compared to GANs, the method maintains the lowest Wasserstein distance, suggesting a regularization effect from minimizing pixel-wise transport cost.
• Theoretical analysis shows that the method minimizes the Wasserstein distance to both the empirical dataset and the underlying population distribution, with bounds that are polynomial in dimension rather than exponential.
• The authors conjecture that minimizing Wasserstein distance via pixel-wise metrics induces a mode-covering bias, and propose combining with perceptual or adversarial losses to improve sharpness in future work.