[Paper Review] FineGAN: Unsupervised Hierarchical Disentanglement for Fine-Grained Object Generation and Discovery

Krishna Kumar Singh, Utkarsh Ojha+1 more|arXiv (Cornell University)|Nov 27, 2018

Advanced Image and Video Retrieval Techniques60 references4 citations

TL;DR

FineGAN is an unsupervised GAN framework that hierarchically disentangles background, object shape, and appearance using information-theoretic constraints on latent codes, enabling realistic fine-grained image generation and unsupervised fine-grained object category discovery. It achieves state-of-the-art clustering performance on CUB, Stanford Dogs, and Stanford Cars datasets, outperforming methods like JULE and DEPICT in NMI and accuracy.

ABSTRACT

We propose FineGAN, a novel unsupervised GAN framework, which disentangles the background, object shape, and object appearance to hierarchically generate images of fine-grained object categories. To disentangle the factors without supervision, our key idea is to use information theory to associate each factor to a latent code, and to condition the relationships between the codes in a specific way to induce the desired hierarchy. Through extensive experiments, we show that FineGAN achieves the desired disentanglement to generate realistic and diverse images belonging to fine-grained classes of birds, dogs, and cars. Using FineGAN's automatically learned features, we also cluster real images as a first attempt at solving the novel problem of unsupervised fine-grained object category discovery. Our code/models/demo can be found at https://github.com/kkanshul/finegan

Motivation & Objective

• To develop an unsupervised generative model that can hierarchically disentangle background, object shape, and appearance without any fine-grained annotations.
• To address the challenge of unsupervised fine-grained object category discovery, a novel problem not previously solved in the literature.
• To learn disentangled representations that capture both structural (shape) and detailed (appearance) factors of variation in fine-grained objects.
• To generate diverse, realistic images of fine-grained categories such as birds, dogs, and cars through hierarchical conditioning on latent codes.
• To demonstrate that features learned from unsupervised hierarchical generation can be used to cluster real images into accurate fine-grained categories.

Proposed method

• FineGAN uses a hierarchical generative process that first generates a background image, then a parent image conditioned on a shape code, and finally a child image conditioned on both the parent code and appearance code.
• It enforces high mutual information between the parent latent code and the parent image, and between the child latent code (conditioned on the parent code) and the child image, using information-theoretic regularization.
• The model conditions on learned masks at both parent and child stages to focus latent codes on relevant object regions and to enable proper image stitching across stages.
• It introduces a hierarchical constraint by grouping child codes under the same parent code, which encourages the parent code to capture shape and the child code to capture appearance variation.
• The model is trained using a GAN objective with adversarial loss and mutual information maximization to ensure disentanglement and realism.
• FineGAN automatically generates segmentation masks at both stages, which guide the generator to attend to object-specific regions and improve disentanglement.

Experimental results

Research questions

• RQ1 Can a generative model with hierarchical disentanglement of background, shape, and appearance generate realistic and diverse fine-grained images without any supervision?
• RQ2 Can the disentangled features learned by such a model be used to cluster real images into fine-grained categories in an unsupervised manner?
• RQ3 How does hierarchical disentanglement compare to flat disentanglement (e.g., InfoGAN) in capturing fine-grained object variations?
• RQ4 To what extent do the learned parent and child codes capture shape and appearance factors respectively, and are they complementary for clustering?
• RQ5 Can the model discover meaningful fine-grained structure even when the true number of categories is unknown?

Key findings

• FineGAN achieves state-of-the-art performance in unsupervised fine-grained object clustering, with 0.403 NMI and 0.126 accuracy on the CUB dataset, significantly outperforming JULE (0.204 NMI, 0.045 accuracy) and DEPICT (0.290 NMI, 0.061 accuracy).
• On the Stanford Dogs dataset, FineGAN achieves 0.233 NMI and 0.079 accuracy, surpassing JULE-ResNet-50 (0.148 NMI, 0.044 accuracy) and DEPICT-Large (0.183 NMI, 0.054 accuracy).
• On the Stanford Cars dataset, FineGAN achieves 0.354 NMI and 0.078 accuracy, outperforming all baselines, including DEPICT-Large (0.330 NMI, 0.062 accuracy).
• Ablation studies show that using only child code features reduces accuracy to 0.017 on birds, confirming that parent and child codes are complementary and capture distinct aspects (shape vs. appearance).
• FineGAN successfully disentangles background, shape, and appearance, as shown by qualitative results where varying only the child code changes appearance while keeping shape and background constant.
• In contrast, InfoGAN fails to disentangle appearance from background, as evidenced by consistent background variation across images with the same child code, indicating background bias in latent code prediction.