[Paper Review] Manifold Topology Divergence: a Framework for Comparing Data Manifolds

Serguei Barannikov, Ilya Trofimov+5 more|arXiv (Cornell University)|Jun 8, 2021

Generative Adversarial Networks and Image Synthesis31 references4 citations

TL;DR

This paper introduces Manifold Topology Divergence (MTop-Divergence), a novel, domain-agnostic framework for comparing data manifolds using a new tool called Cross-Barcode(P,Q) that tracks multiscale topological discrepancies between point clouds. The method scales linearly with ambient dimension and outperforms existing metrics like FID and GScore in detecting mode collapse, mode invention, and manifold distortions across images, 3D shapes, and time-series data.

ABSTRACT

We develop a framework for comparing data manifolds, aimed, in particular, towards the evaluation of deep generative models. We describe a novel tool, Cross-Barcode(P,Q), that, given a pair of distributions in a high-dimensional space, tracks multiscale topology spacial discrepancies between manifolds on which the distributions are concentrated. Based on the Cross-Barcode, we introduce the Manifold Topology Divergence score (MTop-Divergence) and apply it to assess the performance of deep generative models in various domains: images, 3D-shapes, time-series, and on different datasets: MNIST, Fashion MNIST, SVHN, CIFAR10, FFHQ, chest X-ray images, market stock data, ShapeNet. We demonstrate that the MTop-Divergence accurately detects various degrees of mode-dropping, intra-mode collapse, mode invention, and image disturbance. Our algorithm scales well (essentially linearly) with the increase of the dimension of the ambient high-dimensional space. It is one of the first TDA-based practical methodologies that can be applied universally to datasets of different sizes and dimensions, including the ones on which the most recent GANs in the visual domain are trained. The proposed method is domain agnostic and does not rely on pre-trained networks.

Motivation & Objective

• To address the lack of reliable, domain-agnostic evaluation metrics for deep generative models, especially in detecting topological failures like mode dropping and collapse.
• To overcome the limitations of existing metrics such as FID, which rely on pre-trained networks and fail to capture subtle geometric and topological discrepancies in high-dimensional data.
• To develop a scalable topological method that can be applied to diverse data types—including 2D images, 3D point clouds, and time-series—without architectural or domain-specific assumptions.
• To provide a model evaluation tool that reflects the true evolution of data manifold structure during training, enabling insights into training dynamics and early stopping.
• To demonstrate that topology-based metrics can outperform standard metrics in detecting geometric and topological anomalies in generated data.

Proposed method

• Proposes Cross-Barcode(P,Q), a topological tool that encodes multiscale differences in the Betti numbers and persistence diagrams between two point clouds P (real data) and Q (generated data), capturing topological discrepancies across scales.
• Defines Manifold Topology Divergence (MTop-Div) as a divergence score derived from Cross-Barcode(P,Q), quantifying the topological dissimilarity between the manifolds underlying the data distributions.
• Uses persistent homology to compute topological features of both real and generated data point clouds, focusing on connected components (H0), loops (H1), and voids (H2) at multiple scales.
• Applies a modified version of witness complex construction with optimized landmark selection to ensure scalability and reduce stochasticity in barcode computation.
• Employs normalization and distance computation between persistence diagrams using a stable metric (e.g., Wasserstein or bottleneck distance) to derive the final divergence score.
• Scales the method to high-dimensional data (up to D = 10^7) by leveraging efficient computational geometry and parallelization techniques, overcoming prior TDA scalability bottlenecks.

Experimental results

Research questions

• RQ1 Can a topology-based metric detect subtle topological failures in generative models—such as mode dropping, mode collapse, and mode invention—more effectively than existing metrics?
• RQ2 How does the MTop-Divergence score behave during the training of GANs across different data modalities, and does it reflect meaningful changes in the data manifold structure?
• RQ3 To what extent does MTop-Divergence correlate with domain-specific evaluation metrics (e.g., FID, discriminative score) and provide complementary insights into model quality?
• RQ4 Can a topology-based evaluation method be made scalable and practical for modern high-dimensional datasets like FFHQ (D ≈ 10^7) without relying on pre-trained networks?
• RQ5 How does MTop-Divergence compare in sensitivity to geometric transformations (e.g., shift, dilation) compared to existing topology-based metrics like Geometry Score?

Key findings

• MTop-Divergence successfully detects manifold explosions and topological distortions during GAN training on 3D shapes, with a peak at epoch 50 when generated objects diverge significantly from the real manifold, a shift not captured by MMD, JSD, or Coverage.
• On the FFHQ dataset (up to 10^7 dimensions), MTop-Divergence computation scales essentially linearly with ambient dimension, demonstrating practical feasibility for large-scale, high-dimensional data.
• MTop-Divergence correlates well with domain-specific metrics and provides additional insights into training dynamics, such as the emergence of spurious modes or manifold collapse, which are missed by conventional metrics.
• The method outperforms FID, GScore, and other baselines in detecting topological anomalies: for example, it identifies mode dropping and intra-mode collapse where FID fails due to its Gaussian approximation bias.
• In time-series generation with TimeGAN, MTop-Divergence decreases in tandem with the discriminative score, but reveals a more nuanced convergence pattern—detecting when real and generated manifolds become topologically similar at epoch 2000, a signal missed by the discriminative score.
• MTop-Divergence is robust to geometric transformations and does not suffer from the insensitivity issues of GScore, which fails to detect constant shifts or dilations due to its approximate barcode computation on 64 landmarks.