[Paper Review] Unsupervised multi-modal Styled Content Generation

Omry Sendik, Dani Lischinski+1 more|arXiv (Cornell University)|Jan 10, 2020

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes uMM-GAN, a novel unsupervised generative model that learns multiple latent modes in a multi-modal data distribution by introducing K learned root constants, enabling disentangled control over both mode (e.g., animal species) and style (e.g., fur color). The method improves distribution approximation and FID scores over baseline GANs, especially when K matches or exceeds the true number of modes.

ABSTRACT

The emergence of deep generative models has recently enabled the automatic generation of massive amounts of graphical content, both in 2D and in 3D. Generative Adversarial Networks (GANs) and style control mechanisms, such as Adaptive Instance Normalization (AdaIN), have proved particularly effective in this context, culminating in the state-of-the-art StyleGAN architecture. While such models are able to learn diverse distributions, provided a sufficiently large training set, they are not well-suited for scenarios where the distribution of the training data exhibits a multi-modal behavior. In such cases, reshaping a uniform or normal distribution over the latent space into a complex multi-modal distribution in the data domain is challenging, and the generator might fail to sample the target distribution well. Furthermore, existing unsupervised generative models are not able to control the mode of the generated samples independently of the other visual attributes, despite the fact that they are typically disentangled in the training data. In this paper, we introduce UMMGAN, a novel architecture designed to better model multi-modal distributions, in an unsupervised fashion. Building upon the StyleGAN architecture, our network learns multiple modes, in a completely unsupervised manner, and combines them using a set of learned weights. We demonstrate that this approach is capable of effectively approximating a complex distribution as a superposition of multiple simple ones. We further show that UMMGAN effectively disentangles between modes and style, thereby providing an independent degree of control over the generated content.

Motivation & Objective

• To address the limitation of existing GANs in modeling multi-modal data distributions without supervision.
• To enable independent control over the mode of generated samples (e.g., cat vs. dog) separate from style attributes.
• To improve the fidelity and diversity of generated samples by approximating complex data distributions as a superposition of K simpler, learned modes.
• To demonstrate that unsupervised mode discovery via multiple root constants leads to better distribution approximation than single-root GANs.

Proposed method

• The model extends StyleGAN by replacing the single learned constant with K distinct root constants, each serving as a seed for a different mode in the data distribution.
• Each generated sample is formed by combining a weighted mixture of the K mode-specific features, with the weights learned during training.
• The generator uses AdaIN for style control, while mode selection is decoupled via the K root constants, enabling independent manipulation of mode and style.
• The architecture is trained end-to-end using adversarial loss, with no need for class labels or annotations.
• The number of modes K is a hyperparameter, and the model learns to assign samples to the most relevant mode based on data distribution.
• The method implicitly learns mode disentanglement by encouraging each root to capture distinct high-level content characteristics (e.g., species, object pose).

Experimental results

Research questions

• RQ1 Can an unsupervised GAN architecture effectively model multi-modal data distributions without class annotations?
• RQ2 Does learning multiple latent modes via K root constants improve distribution approximation compared to single-root GANs?
• RQ3 Can mode and style be effectively disentangled in a generative model, enabling independent control over high-level content and visual appearance?
• RQ4 How does the number of learned modes K affect the quality of generated samples, as measured by FID score?
• RQ5 Does the model generalize to natural image datasets with complex, real-world multi-modal distributions?

Key findings

• uMM-GAN achieves significantly lower FID scores than baseline StyleGAN on multi-modal datasets, with improvements of up to 1.47 FID points when increasing K from 2 to 3 on a 3-class dataset.
• On the Cats & Dogs dataset, FID score drops from 126.90 (K=1) to 103.49 (K=16), demonstrating consistent improvement with increasing K.
• On the Cars and Bedrooms datasets, FID scores decrease from 167.22 (K=1) to 145.54 (K=16) and from 189.53 (K=1) to 157.01 (K=16), respectively, showing effectiveness on complex natural images.
• The model successfully disentangles mode from style, as shown by samples where mode is switched while style parameters remain fixed, preserving visual attributes like fur color and pattern.
• The model performs best when K ≥ N (true number of modes), with diminishing returns when K ≫ N, suggesting a trade-off in mode capacity.
• The method generalizes well across datasets, including synthetic (letters) and real-world (animals, cars, interiors), indicating robustness to data complexity.