[Paper Review] Contrastive Fine-grained Class Clustering via Generative Adversarial Networks

Yunji Kim, Jung-Woo Ha|arXiv (Cornell University)|Dec 30, 2021

Face recognition and analysis5 citations

TL;DR

C3-GAN proposes a novel GAN-based method for unsupervised fine-grained class clustering by combining InfoGAN's categorical inference with contrastive learning to learn a clustering-friendly embedding space. It achieves state-of-the-art performance on four fine-grained datasets while mitigating mode collapse by training a discriminator to optimize contrastive loss using image-latent pairs as positive and negative samples.

ABSTRACT

Unsupervised fine-grained class clustering is a practical yet challenging task due to the difficulty of feature representations learning of subtle object details. We introduce C3-GAN, a method that leverages the categorical inference power of InfoGAN with contrastive learning. We aim to learn feature representations that encourage a dataset to form distinct cluster boundaries in the embedding space, while also maximizing the mutual information between the latent code and its image observation. Our approach is to train a discriminator, which is also used for inferring clusters, to optimize the contrastive loss, where image-latent pairs that maximize the mutual information are considered as positive pairs and the rest as negative pairs. Specifically, we map the input of a generator, which was sampled from the categorical distribution, to the embedding space of the discriminator and let them act as a cluster centroid. In this way, C3-GAN succeeded in learning a clustering-friendly embedding space where each cluster is distinctively separable. Experimental results show that C3-GAN achieved the state-of-the-art clustering performance on four fine-grained image datasets, while also alleviating the mode collapse phenomenon. Code is available at https://github.com/naver-ai/c3-gan.

Motivation & Objective

• To address the challenge of unsupervised fine-grained class clustering, where subtle visual differences between classes make representation learning difficult.
• To overcome limitations of existing self-supervised learning methods that treat fine-grained variations as noise due to data augmentation.
• To eliminate reliance on human-annotated bounding boxes or classifiers, enabling application to diverse, uncurated datasets.
• To improve GAN training stability by alleviating mode collapse through a contrastive regularization scheme.
• To learn a disentangled, clustering-friendly embedding space where each class forms a distinct, linearly separable cluster.

Proposed method

• C3-GAN uses a conditional GAN framework with two latent codes: a categorical code $c$ for class identity and a continuous code $z$ for content variation.
• The generator maps $c$ and $z$ to a synthesized image, while the discriminator embeds the image and uses the categorical code $c$ to predict cluster centroids.
• Contrastive learning is applied by defining positive pairs as image-latent code pairs with high mutual information and negative pairs as all others.
• The discriminator is trained to maximize contrastive loss, pulling features of images with the same $c$ closer and pushing features with different $c$ apart.
• The method incorporates scene decomposition without annotations, inspired by PerturbGAN, to encourage the generator to focus on foreground objects.
• The training objective regularizes the latent space to form distinct cluster boundaries, improving both clustering and generation quality.

Experimental results

Research questions

• RQ1 Can contrastive learning be effectively combined with InfoGAN to improve unsupervised fine-grained clustering without human-annotated labels?
• RQ2 Does training a discriminator with contrastive loss on image-latent pairs lead to better cluster separation in the embedding space?
• RQ3 Can a GAN-based framework achieve state-of-the-art clustering performance on fine-grained datasets while mitigating mode collapse?
• RQ4 To what extent does scene decomposition without bounding box annotations improve feature learning for fine-grained clustering?
• RQ5 Does the induced embedding space enhance both clustering and image generation quality simultaneously?

Key findings

• C3-GAN achieved state-of-the-art clustering performance on four fine-grained datasets: CUB, Stanford Cars, Stanford Dogs, and Oxford Flowers.
• The method significantly reduced mode collapse compared to baseline GANs, with visual analysis showing diverse intra-class variations in color, shape, and layout.
• Only C3-GAN successfully performed scene decomposition without annotations, while baselines like FineGAN and MixNMatch collapsed into generating only background images.
• C3-GAN achieved competitive FID and Inception Score on image generation, with Inception Score of 8.7 on CUB and 8.6 on Stanford Dogs, indicating high-quality image synthesis.
• Reverse KL scores showed close alignment between real and generated class distributions, indicating effective distribution modeling.
• Ablation studies confirmed that both $c$ and $z$ disentangle class identity and content variation, with $c$ controlling species and $z$ controlling pose and background.