[Paper Review] Unsupervised Object Segmentation by Redrawing

Mickaël Chen, Thierry Artières+1 more|arXiv (Cornell University)|May 27, 2019

Advanced Neural Network ApplicationsComputer Science62 references61 citations

TL;DR

ReDO learns to segment objects in images without labels by enforcing a generative process where objects can be redrawn in place, guided by a GAN discriminator. It yields competitive segmentation masks compared to supervised baselines with limited labeled data.

ABSTRACT

Object segmentation is a crucial problem that is usually solved by using supervised learning approaches over very large datasets composed of both images and corresponding object masks. Since the masks have to be provided at pixel level, building such a dataset for any new domain can be very time-consuming. We present ReDO, a new model able to extract objects from images without any annotation in an unsupervised way. It relies on the idea that it should be possible to change the textures or colors of the objects without changing the overall distribution of the dataset. Following this assumption, our approach is based on an adversarial architecture where the generator is guided by an input sample: given an image, it extracts the object mask, then redraws a new object at the same location. The generator is controlled by a discriminator that ensures that the distribution of generated images is aligned to the original one. We experiment with this method on different datasets and demonstrate the good quality of extracted masks.

Motivation & Objective

• Motivate unsupervised segmentation by modeling a scene as independent regions that can be swapped without disrupting realism.
• Propose ReDO, an adversarial framework where a mask extractor and per-region generators learn to reconstruct realistic images by redrawing objects.
• Impose constraints to avoid trivial solutions (e.g., empty or all-in-one-region masks) and preserve information in drawn regions.

Proposed method

• Define a three-step generative process: composition of regions with masks M^k, independent drawing of object pixels V^k, and assembling the final image I.
• Learn F (mask extractor) and G_k (region generators) via a GAN objective with a discriminator D enforcing fidelity to the real data distribution.
• Redraws are performed one region at a time during training to prevent trivial solutions and enforce meaningful segmentation.
• Enforce information conservation by ensuring the latent z_i associated with a region can be recovered from the output, similar to InfoGAN ideas.
• Use hinge GAN losses and an adversarial learning loop to optimize G_F (the combined generator) and D.
• Architectures: F based on PSPNet-like encoder with pyramid pooling; G_k and D inspired by SAGAN with spectral normalization and self-attention; conditional batch normalization in G_k to encode texture/color.
• Training stability: manage potential collapse to empty masks with hyperparameter tuning and occasional restarts.

Experimental results

Research questions

• RQ1 Can fully unsupervised learning discover meaningful object segmentation masks by modeling a generative process that swaps object content while preserving distribution?
• RQ2 How well can a mask extractor F and per-object generators G_k learn to redraw objects such that the reconstructed images remain indistinguishable from real images?
• RQ3 What constraints are necessary to prevent trivial solutions (e.g., empty masks or all pixels in one region) and ensure information about drawn regions is preserved?
• RQ4 How does ReDO perform on real-world datasets (LFW, Flowers, CUB) compared to supervised baselines with limited labeled data, and can it handle multi-class/unknown objects?
• RQ5 Can the unsupervised masks generalize to combined datasets (e.g., Flowers+LFW) without labels?

Key findings

• ReDO produces non-trivial object masks that enable meaningful region-wise redrawing across datasets.
• On LFW, ReDO achieves 0.917 accuracy and 0.781 IoU on the test set, without using labels for training.
• On CUB-200-2011, ReDO achieves 0.845 test accuracy and 0.426 IoU, with 0.840 train accuracy and 0.423 train IoU.
• On Flowers, ReDO reaches 0.879 test accuracy and 0.764 IoU (Flowers*) with 0.886 train accuracy and 0.780 train IoU.
• On a fused Flowers+LFW dataset, ReDO achieves 0.856 test accuracy and 0.691 IoU without labels.
• Compared to supervised baselines with limited labeled data, the unsupervised model attains comparable performance and can match supervised models trained with around 50–60 labeled examples on LFW.
• Qualitative results show that the inferred masks align with object-like regions and that redrawn objects preserve target textures/colors via the latent code z_i.