[Paper Review] PUG: Photorealistic and Semantically Controllable Synthetic Data for Representation Learning

Florian Bordes, Shashank Shekhar+4 more|arXiv (Cornell University)|Aug 8, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces PUG (Photorealistic Unreal Graphics), a new family of synthetic datasets generated using the Unreal Engine to enable photorealistic, semantically controllable data for representation learning. By combining high realism with precise control over scene factors like pose, texture, and lighting, PUG enables rigorous evaluation of vision models' robustness and generalization, demonstrating significant improvements in zero-shot transfer and failure mode analysis over existing benchmarks.

ABSTRACT

Synthetic image datasets offer unmatched advantages for designing and evaluating deep neural networks: they make it possible to (i) render as many data samples as needed, (ii) precisely control each scene and yield granular ground truth labels (and captions), (iii) precisely control distribution shifts between training and testing to isolate variables of interest for sound experimentation. Despite such promise, the use of synthetic image data is still limited -- and often played down -- mainly due to their lack of realism. Most works therefore rely on datasets of real images, which have often been scraped from public images on the internet, and may have issues with regards to privacy, bias, and copyright, while offering little control over how objects precisely appear. In this work, we present a path to democratize the use of photorealistic synthetic data: we develop a new generation of interactive environments for representation learning research, that offer both controllability and realism. We use the Unreal Engine, a powerful game engine well known in the entertainment industry, to produce PUG (Photorealistic Unreal Graphics) environments and datasets for representation learning. In this paper, we demonstrate the potential of PUG to enable more rigorous evaluations of vision models.

Motivation & Objective

• To address the limitations of existing synthetic datasets, which lack realism and fine-grained control, by creating a new generation of photorealistic, controllable synthetic data for representation learning.
• To enable systematic evaluation of vision models' robustness to controlled distribution shifts in pose, texture, lighting, and background.
• To provide a scalable, accessible, and realistic alternative to real-world datasets that face issues with privacy, bias, and copyright.
• To demonstrate the utility of synthetic data in overcoming known limitations of vision-language benchmarks, such as background sensitivity and poor generalization.

Proposed method

• Leveraging the Unreal Engine, a high-fidelity game engine known for photorealism, to generate PUG environments and datasets with precise control over scene factors.
• Designing four distinct datasets: PUG: Animals for OOD generalization, PUG: ImageNet for robustness evaluation, PUG: SPAR for vision-language model probing, and PUG: AR4T for fine-tuning.
• Implementing the TorchMultiverse Python library to enable interactive, programmatic dataset creation with fine-grained control over scene attributes.
• Using pre-trained CLIP models and fine-tuning them on PUG datasets to evaluate zero-shot and fine-tuned performance across various factor shifts.
• Introducing a single-environment setup (salt flats) to isolate and analyze model failures due to background changes, independent of background variation.
• Applying factor-labeling techniques to enable probing of representation learning dynamics, such as which factors models attend to during training.

Experimental results

Research questions

• RQ1 To what extent can photorealistic synthetic data improve the robustness and generalization of vision models compared to real-world benchmarks?
• RQ2 How do vision-language models perform under controlled variations in object texture, position, and background, and what failure modes emerge?
• RQ3 Can synthetic data be used to systematically evaluate and improve model invariance to specific factors such as pose, lighting, and object size?
• RQ4 How does fine-tuning on synthetic data from PUG: AR4T improve performance on downstream vision-language tasks compared to real-world benchmarks?
• RQ5 What insights can be gained about model attention and representation learning dynamics by using datasets with precise factor controls?

Key findings

• PUG: SPAR revealed that vision-language models are highly sensitive to background changes, with retrieval accuracy dropping from 94% on a single background (salt flats) to 78% across multiple backgrounds.
• Even large vision-language models like ViT-G-14 (OpenCLIP) failed on simple PUG: SPAR examples, with zero-shot accuracy dropping to 48.44% under texture changes and below 30% for positional reasoning tasks.
• Fine-tuning CLIP models on PUG: AR4T for 10 epochs on a 200K dataset and 2 epochs on a 1M dataset led to significant performance gains, demonstrating the reliability and utility of synthetic data for fine-tuning.
• The PUG: ImageNet dataset enabled systematic evaluation of model robustness across 100+ factors, including pose, lighting, and texture, revealing significant performance drops under distribution shifts.
• The PUG: Animals dataset enabled effective probing of foundation model representations, showing that models can learn disentangled representations when trained on controlled, realistic data.
• The TorchMultiverse library enabled efficient, programmatic generation of diverse, factor-controlled data, proving essential for scalable and reproducible representation learning experiments.