[Paper Review] Combating Mode Collapse in GANs via Manifold Entropy Estimation

Haozhe Liu, Bing Li+6 more|arXiv (Cornell University)|Aug 25, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes MaEM-GAN, a novel GAN training pipeline that combats mode collapse by generalizing the discriminator to learn a structured embedding space and maximizing the entropy of generated samples within it. By introducing manifold regularization (DLLE and DIsoMap) and a replay-buffer-based non-parametric entropy estimator, the method improves sample diversity without sacrificing generation quality, outperforming state-of-the-art GANs and energy-based models on benchmark datasets.

ABSTRACT

Generative Adversarial Networks (GANs) have shown compelling results in various tasks and applications in recent years. However, mode collapse remains a critical problem in GANs. In this paper, we propose a novel training pipeline to address the mode collapse issue of GANs. Different from existing methods, we propose to generalize the discriminator as feature embedding and maximize the entropy of distributions in the embedding space learned by the discriminator. Specifically, two regularization terms, i.e., Deep Local Linear Embedding (DLLE) and Deep Isometric feature Mapping (DIsoMap), are designed to encourage the discriminator to learn the structural information embedded in the data, such that the embedding space learned by the discriminator can be well-formed. Based on the well-learned embedding space supported by the discriminator, a non-parametric entropy estimator is designed to efficiently maximize the entropy of embedding vectors, playing as an approximation of maximizing the entropy of the generated distribution. By improving the discriminator and maximizing the distance of the most similar samples in the embedding space, our pipeline effectively reduces the mode collapse without sacrificing the quality of generated samples. Extensive experimental results show the effectiveness of our method, which outperforms the GAN baseline, MaF-GAN on CelebA (9.13 vs. 12.43 in FID) and surpasses the recent state-of-the-art energy-based model on the ANIME-FACE dataset (2.80 vs. 2.26 in Inception score). The code is available at https://github.com/HaozheLiu-ST/MEE

Motivation & Objective

• To address the persistent problem of mode collapse in GANs, which limits the diversity of generated samples despite high-quality outputs.
• To improve training stability and diversity in GANs by redefining the discriminator’s role from a binary classifier to a feature embedder that preserves data manifold structure.
• To maximize the entropy of the generated distribution indirectly by optimizing the entropy in the discriminator’s learned embedding space.
• To develop an efficient, non-parametric entropy estimation method using a replay buffer to scale to high-dimensional data without intractable partition function computation.
• To demonstrate that entropy maximization in a well-structured embedding space can surpass both standard GANs and recent energy-based models in diversity and fidelity.

Proposed method

• Generalize the discriminator to output a deep feature embedding instead of a scalar, transforming it into a feature embedder that maps real and generated images into an m-dimensional space.
• Introduce Deep Local Linear Embedding (DLLE) and Deep Isometric feature Mapping (DIsoMap) as regularization terms to preserve local and global manifold structures in the embedding space.
• Design a replay-buffer-based non-parametric entropy estimator (RB-MaEM) to efficiently approximate and maximize the entropy of the embedding distribution.
• Formulate the training objective as maximizing the entropy of the embedding vectors, which acts as a proxy for maximizing the entropy of the generated data distribution.
• Integrate the entropy maximization objective into the GAN training loop, maintaining adversarial loss while promoting diversity through the embedding space.
• Apply the method to standard GAN architectures (e.g., StyleGAN-V2, BigGAN) with minimal architectural changes, demonstrating generalization across deep and complex models.

Experimental results

Research questions

• RQ1 Can maximizing the entropy of the generated distribution in a learned embedding space effectively reduce mode collapse in GANs?
• RQ2 How can the discriminator be restructured to preserve the intrinsic manifold structure of data to support reliable entropy estimation?
• RQ3 Can a non-parametric entropy estimator based on a replay buffer efficiently and accurately estimate entropy in high-dimensional embedding spaces?
• RQ4 Does entropy maximization in the embedding space lead to improved sample diversity without degrading image quality compared to standard GANs?
• RQ5 Can this approach surpass state-of-the-art energy-based models in both diversity and fidelity metrics on benchmark datasets?

Key findings

• MaEM-GAN achieves an FID of 9.13 on CelebA, significantly outperforming the MaF-GAN baseline (12.43), demonstrating superior diversity and quality.
• On the ANIMEFACE dataset, MaEM-GAN achieves an Inception Score of 2.80, surpassing the recent state-of-the-art energy-based model (EBM-BB) with 2.26.
• The I-Variance score increases from 3.41 to 4.88 when RB-MaEM is included, proving its effectiveness in enhancing sample diversity.
• The method consistently improves training stability and performance when applied to deep architectures like StyleGAN-V2 and BigGAN.
• Ablation studies confirm that both DLLE and DIsoMap are essential for forming a well-structured embedding space that enables effective entropy maximization.
• Visualization of top-ranked samples across embedding dimensions shows that the discriminator learns to assess realness from diverse attributes (e.g., color, style), confirming the method’s ability to capture multi-aspect data variation.