[Paper Review] The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation

Thibault Séjourné, François‐Xavier Vialard+1 more|arXiv (Cornell University)|Sep 9, 2020

Topological and Geometric Data AnalysisComputer Science47 references34 citations

TL;DR

Introduces two unbalanced Gromov–Wasserstein formulations (UGW divergence and CGW distance) to compare metric measure spaces with arbitrary positive measures, and provides efficient GPU-friendly algorithms with theoretical guarantees.

ABSTRACT

Comparing metric measure spaces (i.e. a metric space endowed with aprobability distribution) is at the heart of many machine learning problems. The most popular distance between such metric measure spaces is theGromov-Wasserstein (GW) distance, which is the solution of a quadratic assignment problem. The GW distance is however limited to the comparison of metric measure spaces endowed with a probability distribution. To alleviate this issue, we introduce two Unbalanced Gromov-Wasserstein formulations: a distance and a more tractable upper-bounding relaxation.They both allow the comparison of metric spaces equipped with arbitrary positive measures up to isometries. The first formulation is a positive and definite divergence based on a relaxation of the mass conservation constraint using a novel type of quadratically-homogeneous divergence. This divergence works hand in hand with the entropic regularization approach which is popular to solve large scale optimal transport problems. We show that the underlying non-convex optimization problem can be efficiently tackled using a highly parallelizable and GPU-friendly iterative scheme. The second formulation is a distance between mm-spaces up to isometries based on a conic lifting. Lastly, we provide numerical experiments onsynthetic examples and domain adaptation data with a Positive-Unlabeled learning task to highlight the salient features of the unbalanced divergence and its potential applications in ML.

Motivation & Objective

• Motivate the need for comparing mm-spaces without requiring probability measures (mass variations and outliers).
• Define two unbalanced formulations that extend GW to arbitrary positive measures up to isometries.
• Establish theoretical properties, including definiteness and bounding relationships between UGW and CGW.
• Develop efficient numerical schemes leveraging entropic regularization and Sinkhorn iterations for large-scale problems.
• Illustrate the approaches with synthetic and domain adaptation experiments including positive-unlabeled learning tasks.

Proposed method

• Define Unbalanced GW (UGW) as an infimum of a sum of a transport term and quadratic ϕ-divergence penalties on marginals, yielding a 2-homogeneous divergence.
• Introduce quadratic tensorized ϕ-divergences Dϕ⊗ to compare marginals in the unbalanced setting.
• Provide a reformulation splitting UGW into a transport-like term Lc and mass-creation/destruction terms, enabling analysis (Lemma 1).
• Present a Conic Gromov-Wasserstein (CGW) distance built on lifting to a cone and using a cone distance Д on 𝔠[ℝ+], yielding a distance up to isometries (Theorem 1).
• Show CGW is bounded above by UGW (CGW ≤ UGW) and that CGW is a distance when Д is a metric on the cone.
• Prove existence of minimizers for UGW and CGW under standard conditions (Propositions 1, 3).
• Develop a bi-convex, entropically regularized relaxation of UGW (UGWε) and alternate Sinkhorn-type optimization (Algorithm 1).
• Provide a corresponding relaxation for CGW and discuss tightness results (Theorem 3).

Experimental results

Research questions

• RQ1 Can mm-spaces with arbitrary positive measures be meaningfully compared up to isometries without enforcing mass conservation?
• RQ2 How can we define a computationally tractable unbalanced analogue of GW that leverages entropic regularization?
• RQ3 What is the relation between the unbalanced GW divergence (UGW) and the conic GW distance (CGW), and can UGW upper-bound CGW?
• RQ4 Under what conditions is the bi-convex, entropically regularized relaxation tight for GW and CGW?
• RQ5 Do these unbalanced formulations perform well in practice on synthetic data and domain adaptation tasks such as positive-unlabeled learning?

Key findings

• UGW provides a 2-homogeneous divergence that relaxes mass conservation and allows comparison of mm-spaces with arbitrary positive measures.
• CGW defines a true distance between mm-spaces up to isometries, and CGW is bounded above by UGW (CGW ≤ UGW).
• Existence of minimizers for both UGW and CGW is established under practical conditions (compact spaces, appropriate φ-divergences, and λ).
• A bi-convex, entropically regularized relaxation (UGWε) yields a GPU-friendly optimization scheme with convergence properties and ground for back-propagation in ML tasks.
• For certain kernels (e.g., squared distance), the bi-convex relaxations are tight, ensuring recovered solutions are optimal for the original problems.
• Numerical experiments demonstrate robustness to mass imbalances and alignment with domain adaptation and positive-unlabeled learning tasks, outperforming or matching relevant competitors in provided settings.