[Paper Review] Syn2Real: A New Benchmark forSynthetic-to-Real Visual Domain Adaptation

Xingchao Peng, Ben Usman+4 more|arXiv (Cornell University)|Jun 26, 2018

Domain Adaptation and Few-Shot Learning44 references49 citations

TL;DR

The paper introduces the Syn2Real benchmark for unsupervised synthetic-to-real domain adaptation across closed-set classification, open-set classification, and object detection, showing large gaps in performance and proposing a challenging multi-task evaluation setup.

ABSTRACT

Unsupervised transfer of object recognition models from synthetic to real data is an important problem with many potential applications. The challenge is how to "adapt" a model trained on simulated images so that it performs well on real-world data without any additional supervision. Unfortunately, current benchmarks for this problem are limited in size and task diversity. In this paper, we present a new large-scale benchmark called Syn2Real, which consists of a synthetic domain rendered from 3D object models and two real-image domains containing the same object categories. We define three related tasks on this benchmark: closed-set object classification, open-set object classification, and object detection. Our evaluation of multiple state-of-the-art methods reveals a large gap in adaptation performance between the easier closed-set classification task and the more difficult open-set and detection tasks. We conclude that developing adaptation methods that work well across all three tasks presents a significant future challenge for syn2real domain transfer.

Motivation & Objective

• Motivate unsupervised transfer from synthetic to real imagery for robust object recognition.
• Provide a large-scale, multi-task benchmark (classification and detection) with fixed synthetic source and two real target domains.
• Compare state-of-the-art UDA methods across tasks and analyze why adaptation succeeds or fails under different settings.
• Offer open-source data, code, and 3D models to spur progress in syn-to-real domain transfer.

Proposed method

• Create a synthetic source domain from 3D CAD models for 12 object categories.
• Construct two real target domains: MS COCO cropped images for validation and YouTube Bounding Boxes for testing.
• Define three tasks (Syn2Real-C closed-set classification, Syn2Real-O open-set classification, Syn2Real-D detection) under unsupervised domain adaptation.
• Evaluate baselines (DAN, Deep CORAL) and leading challenge solutions (Self-Ensembling SE).
• Analyze factors affecting adaptation (synthetic data volume, target image size, synthetic background, pretraining).
• Provide public dataset, benchmarks, and open-source rendering pipeline and 3D models.

Experimental results

Research questions

• RQ1 How well do existing unsupervised domain adaptation methods transfer from a synthetic source to real targets across classification and detection tasks?
• RQ2 What are the relative difficulties and gaps between closed-set, open-set, and detection under synthetic-to-real transfer?
• RQ3 How do factors like data quantity, image resolution, and background realism influence adaptation performance?

Key findings

• UDA methods improve over source-only performance, e.g., Deep CORAL and DAN raise validation accuracy from 28.1% to 45.5% and 51.6% respectively for Syn2Real-C.
• Open-set and detection tasks remain significantly more challenging than closed-set classification, with larger gaps to in-domain or oracle performance.
• The SE (Self-Ensembling) method achieved the top challenge result, substantially boosting performance (e.g., from 45.3% to 91.8% on the source model for Syn2Real-C).
• Pretraining on ImageNet substantially boosts adaptation, whereas training from scratch (SE*) performs poorly without sufficient synthetic data.
• The detection task shows that adaptation methods that work for classification do not directly transfer to end-to-end detectors, indicating a need for new approaches.