[Paper Review] Cycle Consistent Probability Divergences Across Different Spaces

Zhengxin Zhang, Youssef Mroueh+2 more|arXiv (Cornell University)|Nov 22, 2021

Generative Adversarial Networks and Image Synthesis37 references4 citations

TL;DR

This paper introduces the unbalanced bidirectional Gromov-Monge (UBGM) divergence, a novel discrepancy measure for probability distributions on different metric spaces that enforces cycle-consistent, isometry-preserving mappings via two pushforward maps. It kernelizes UBGM into the generalized maximum mean discrepancy (GMMD), proving convergence rates and demonstrating generalization to unseen data, thereby providing a theoretical foundation for cycle GANs and enabling efficient, scalable matching across heterogeneous spaces.

ABSTRACT

Discrepancy measures between probability distributions are at the core of statistical inference and machine learning. In many applications, distributions of interest are supported on different spaces, and yet a meaningful correspondence between data points is desired. Motivated to explicitly encode consistent bidirectional maps into the discrepancy measure, this work proposes a novel unbalanced Monge optimal transport formulation for matching, up to isometries, distributions on different spaces. Our formulation arises as a principled relaxation of the Gromov-Haussdroff distance between metric spaces, and employs two cycle-consistent maps that push forward each distribution onto the other. We study structural properties of the proposed discrepancy and, in particular, show that it captures the popular cycle-consistent generative adversarial network (GAN) framework as a special case, thereby providing the theory to explain it. Motivated by computational efficiency, we then kernelize the discrepancy and restrict the mappings to parametric function classes. The resulting kernelized version is coined the generalized maximum mean discrepancy (GMMD). Convergence rates for empirical estimation of GMMD are studied and experiments to support our theory are provided.

Motivation & Objective

• To address the challenge of measuring discrepancy between probability distributions supported on different, incompatible metric spaces.
• To develop a computationally efficient, generalizable discrepancy measure that explicitly encodes consistent bidirectional mappings between data points.
• To provide a theoretical framework for cycle GANs by formalizing cycle-consistent mappings as a principled optimal transport formulation.
• To kernelize the discrepancy for practical use with parametric models like neural networks, enabling scalable estimation.
• To establish convergence rates for empirical GMMD estimation and validate generalization on unseen data.

Proposed method

• Proposes the unbalanced bidirectional Gromov-Monge (UBGM) divergence as a relaxation of the Gromov-Hausdorff distance, using two cycle-consistent maps to push each distribution onto the other space.
• Imposes cycle consistency by requiring the composition of the two maps to approximately preserve the metric structure, ensuring consistent correspondences.
• Restricts the mappings to parametric function classes (e.g., neural networks) and kernelizes the UBGM to derive the generalized maximum mean discrepancy (GMMD).
• Derives convergence rates for two-sample empirical estimation of GMMD under regularity conditions on the kernel and function class.
• Employs amortized inference: trains on a subset of data and evaluates the learned maps on unseen data points to assess generalization.
• Uses marginal MMD and a discrepancy measure Δ to quantitatively evaluate the quality of pushforward maps on test data.

Experimental results

Research questions

• RQ1 Can a principled discrepancy measure be formulated that explicitly encodes bidirectional, cycle-consistent mappings between distributions on different metric spaces?
• RQ2 How can the Gromov-Wasserstein distance be relaxed to enable efficient computation and generalization to new data points?
• RQ3 To what extent does the proposed GMMD framework generalize to unseen data after training on a limited dataset?
• RQ4 What is the theoretical convergence rate of empirical GMMD estimation under standard smoothness and kernel assumptions?
• RQ5 How does the proposed framework relate to and explain the success of cycle GANs in practice?

Key findings

• The proposed GMMD framework successfully generalizes to unseen data, as demonstrated by low marginal MMDs (e.g., <0.002) and small Δ values (e.g., <0.1) on 8000 held-out test points after training on only 4000 points.
• For the heart and embedded heart dataset, GMMD achieved a GMMD value of 0.00711 at λ=0.064, with MMD_X and MMD_Y both below 0.002, indicating strong marginal matching.
• The method maintains low discrepancy (Δ < 0.1) across all tested λ values in amortization experiments, confirming robustness and consistency of the learned maps.
• Convergence rates for empirical GMMD estimation are established under standard assumptions, supporting reliable statistical inference.
• The UBGM divergence is shown to generalize the cycle GAN framework, providing a theoretical explanation for its empirical success.
• Experiments confirm that the kernelized GMMD version enables efficient, scalable matching across different spaces with strong generalization and low computational cost.