[Paper Review] DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Yuxuan Zhang, Huan Ling+6 more|arXiv (Cornell University)|Apr 13, 2021

Advanced Neural Network Applications53 references17 citations

TL;DR

DatasetGAN generates high-quality, detailed semantic segmentation datasets by leveraging a pre-trained StyleGAN to synthesize images and a lightweight decoder to predict pixel-wise labels from only a few human-annotated examples. The method produces synthetic datasets that enable state-of-the-art performance in semantic segmentation and keypoint detection, matching fully supervised models with up to 100× less labeled data.

ABSTRACT

We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort. Current deep networks are extremely data-hungry, benefiting from training on large-scale datasets, which are time consuming to annotate. Our method relies on the power of recent GANs to generate realistic images. We show how the GAN latent code can be decoded to produce a semantic segmentation of the image. Training the decoder only needs a few labeled examples to generalize to the rest of the latent space, resulting in an infinite annotated dataset generator! These generated datasets can then be used for training any computer vision architecture just as real datasets are. As only a few images need to be manually segmented, it becomes possible to annotate images in extreme detail and generate datasets with rich object and part segmentations. To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts. Our approach outperforms all semi-supervised baselines significantly and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as our method.

Motivation & Objective

• To address the high cost and labor intensity of creating large-scale, finely annotated image datasets for semantic segmentation.
• To reduce the need for extensive human annotation by leveraging generative models to synthesize both images and their corresponding semantic labels.
• To enable training of deep learning models on synthetic datasets that match or exceed performance of fully supervised models trained on real data.
• To support complex downstream applications such as 3D reconstruction and animatable object modeling through rich part-level and keypoint annotations.
• To demonstrate that only 16–30 human-annotated examples are sufficient to train models that generalize well across diverse segmentation tasks.

Proposed method

• Leverage a pre-trained StyleGAN to generate diverse, realistic images across the latent space.
• Manually annotate only a small subset (e.g., 16–30) of synthesized images with dense pixel-wise segmentation labels.
• Train a lightweight, shallow decoder network to map the GAN’s latent code to semantic segmentation masks, generalizing across the entire latent space.
• Use the trained decoder to automatically generate an infinite stream of synthetic, fully annotated image-segmentation pairs.
• Train any downstream computer vision model on the synthetic dataset and evaluate on real test data.
• Extend the framework to predict 3D part segmentation and 3D keypoints by integrating differentiable rendering and 2D supervision.

Experimental results

Research questions

• RQ1 Can a small number of human-annotated images from a GAN-generated distribution be used to train a decoder that generalizes to the entire latent space for high-quality semantic segmentation?
• RQ2 To what extent can synthetic datasets generated via this method match or surpass the performance of fully supervised models trained on real data?
• RQ3 Can the method produce rich, fine-grained annotations (e.g., 34 face parts, 32 car parts) with minimal human effort?
• RQ4 How effective is the approach in downstream tasks such as keypoint detection and 3D reconstruction from monocular images?
• RQ5 Can the method support complex 3D applications like animatable 3D asset generation with part-level and keypoint supervision?

Key findings

• DatasetGAN significantly outperforms all semi-supervised learning baselines on 7 segmentation tasks, including face and car part segmentation.
• The method achieves performance on par with fully supervised models, even when using only 1/100th of the annotated data in some cases.
• On the CUB bird dataset, the model trained on synthetic data with 30 human-annotated examples outperformed a fine-tuned baseline using the same annotation budget.
• For car keypoint detection, the model trained on synthetic data with only 16 labeled examples achieved strong performance on 20 real test images.
• The approach enabled the first 3D reconstruction of animatable 3D cars from single images, with rigged wheels, correct materials, and dynamic lighting, using only 30 human-annotated examples.
• The framework successfully generated high-quality, detailed part-level and keypoint annotations, enabling realistic 3D asset creation with physics and animation support.