[Paper Review] Learning from Simulated and Unsupervised Images through Adversarial Training

Ashish Shrivastava, Tomas Pfister+4 more|arXiv (Cornell University)|Dec 22, 2016

Domain Adaptation and Few-Shot Learning7 citations

TL;DR

This paper proposes Simulated+Unsupervised (S+U) learning via SimGAN, a GAN-based method that refines synthetic images using unlabeled real data to improve realism while preserving annotations. By combining a local adversarial loss, self-regularization, and a history-based discriminator update, the method achieves state-of-the-art performance in gaze and hand pose estimation without any labeled real data.

ABSTRACT

With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

Motivation & Objective

• To address the domain gap between synthetic and real image distributions that hinders model generalization.
• To enable training of robust deep learning models on synthetic data without requiring costly human annotations on real images.
• To preserve semantic annotations (e.g., gaze direction, keypoint locations) during image refinement.
• To stabilize GAN training and prevent artifacts when refining synthetic images for downstream tasks.

Proposed method

• A refiner network is trained to transform synthetic images into more realistic versions using adversarial training.
• A local adversarial loss is applied over small image patches to improve realism and reduce artifacts compared to global image-level losses.
• A self-regularization loss penalizes large pixel-wise differences between original synthetic and refined images to preserve annotations.
• The discriminator is updated using a moving average of refined images from previous training steps to stabilize training and prevent mode collapse.
• The refiner and discriminator networks are trained in an alternating fashion with a modified GAN objective.
• A fully convolutional architecture is used to maintain spatial structure and avoid global distortions.

Experimental results

Research questions

• RQ1 Can adversarial training effectively improve the realism of synthetic images while preserving their semantic annotations?
• RQ2 How does using a local adversarial loss compare to a global one in reducing visual artifacts during image refinement?
• RQ3 Does maintaining a history of refined images during discriminator updates improve training stability and image quality?
• RQ4 Can models trained on refined synthetic images outperform those trained on real data without any human-annotated real data?
• RQ5 To what extent does the proposed method improve performance on downstream tasks like gaze estimation and hand pose estimation?

Key findings

• The SimGAN method achieved a 72.4% accuracy at 5-pixel distance for hand pose estimation on the NYU test set when training on refined synthetic data, outperforming real data training (74.5%) and synthetic data training (69.7%).
• With three times more synthetic data, the refined synthetic data achieved 83.3% accuracy at 5-pixel distance, significantly outperforming real data training.
• The gaze estimation error was reduced from 12.2 degrees (without history) to 7.8 degrees (with history) when using a history of refined images during discriminator updates.
• User studies and qualitative evaluations confirmed that refined images were perceived as significantly more realistic than original synthetic images.
• The local adversarial loss effectively eliminated depth boundary artifacts present in images refined with a global adversarial loss.
• The method achieved state-of-the-art performance on the MPIIGaze dataset without any labeled real data, demonstrating the effectiveness of S+U learning.