[Paper Review] Finding an Unsupervised Image Segmenter in Each of Your Deep Generative Models

Luke Melas-Kyriazi, Christian Rupprecht+2 more|arXiv (Cornell University)|May 17, 2021

Generative Adversarial Networks and Image SynthesisComputer Science59 references20 citations

TL;DR

This paper proposes an unsupervised method to automatically discover foreground-background segmentation directions in the latent space of pre-trained deep generative models, such as GANs, without human supervision. By learning a universal latent direction that alters image brightness and consistency, the method generates synthetic segmentation masks to train a segmentation network, achieving state-of-the-art unsupervised performance across diverse datasets including CUB200 and Oxford Flowers.

ABSTRACT

Recent research has shown that numerous human-interpretable directions exist in the latent space of GANs. In this paper, we develop an automatic procedure for finding directions that lead to foreground-background image separation, and we use these directions to train an image segmentation model without human supervision. Our method is generator-agnostic, producing strong segmentation results with a wide range of different GAN architectures. Furthermore, by leveraging GANs pretrained on large datasets such as ImageNet, we are able to segment images from a range of domains without further training or finetuning. Evaluating our method on image segmentation benchmarks, we compare favorably to prior work while using neither human supervision nor access to the training data. Broadly, our results demonstrate that automatically extracting foreground-background structure from pretrained deep generative models can serve as a remarkably effective substitute for human supervision.

Motivation & Objective

• To develop an unsupervised, generator-agnostic method for discovering meaningful latent directions in GANs that enable foreground-background image separation.
• To train a segmentation model using only synthetic, GAN-generated masks without access to real training data or human annotations.
• To demonstrate that high-quality segmentation can be achieved across diverse image domains using a single, pre-trained generator, even when the training and test data distributions differ significantly.
• To show that segmentation performance correlates with GAN quality, suggesting that foreground/background structure is implicitly learned during image generation.
• To establish a new paradigm where concepts like saliency and object separation can be automatically extracted from generative models to replace human supervision in vision tasks.

Proposed method

• The method uses a probing scheme to learn a fixed, global latent offset that induces changes in image appearance, specifically targeting brightness and consistency to isolate foreground and background regions.
• It formulates an optimization objective that balances image brightness change and structural consistency, using a trade-off hyperparameter λ to control the trade-off between these two factors.
• The method identifies two complementary latent directions: one that makes the foreground lighter (vl) and another that makes it darker (vb), both derived from the same optimization process.
• It generates a large-scale synthetic dataset of image-mask pairs using the GAN and the discovered latent directions, which are then used to train a standard segmentation network in a fully supervised manner.
• The approach is fully automatic and does not require manual labeling or model-specific tuning, making it applicable to a wide range of GAN architectures.
• The final segmentation model is evaluated on real-world benchmarks without further fine-tuning, demonstrating generalization across domains.

Experimental results

Research questions

• RQ1 Can a universal latent direction in a GAN's latent space be automatically discovered to enable foreground-background image separation without human supervision?
• RQ2 To what extent can a segmentation model trained on synthetic data generated from a pre-trained GAN generalize to real-world, diverse image datasets?
• RQ3 Does the performance of the resulting segmentation model correlate with the quality of the underlying GAN, suggesting that foreground/background structure is implicitly encoded during training?
• RQ4 Can this method outperform existing unsupervised or handcrafted saliency detection methods without using any real training labels or data?
• RQ5 Can the same latent direction be used across multiple GAN architectures, including those trained on different datasets and resolutions, to achieve consistent segmentation performance?

Key findings

• The method achieves state-of-the-art unsupervised image segmentation performance on benchmarks such as CUB200 and Oxford Flowers, outperforming prior unsupervised approaches without human supervision.
• All 12 evaluated GANs, including those trained on TinyImageNet, produce reasonable segmentation results, demonstrating the method’s broad applicability across diverse generator architectures.
• The best-performing GAN, BigBiGAN, achieves the highest segmentation accuracy, and a strong positive correlation is observed between FID scores (a measure of GAN quality) and segmentation performance.
• Using both the foreground-lighter (vl) and foreground-darker (vb) latent directions together yields performance on par with ensembling two separate models, but with minimal additional computational cost.
• The method achieves competitive results on saliency detection benchmarks, often matching or exceeding supervised and handcrafted methods, despite using no real training data or labels.
• The ablation study confirms that λ = 0.2 yields optimal trade-off between brightness change and consistency, and that the two directions are not exact antipodes due to the nonlinearity of the generator function.