[Paper Review] Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery

Zhongzheng Ren, Yong Jae Lee|arXiv (Cornell University)|Nov 24, 2017

Domain Adaptation and Few-Shot Learning74 references4 citations

TL;DR

This paper proposes a cross-domain self-supervised multi-task network that learns generalizable visual representations from synthetic RGB images by jointly predicting depth, surface normal, and instance contours, while using adversarial feature-space domain adaptation to bridge the domain gap to real images. The method achieves state-of-the-art transfer learning performance on PASCAL VOC 2007 classification and detection, outperforming single-task baselines and competitive methods.

ABSTRACT

In human learning, it is common to use multiple sources of information jointly. However, most existing feature learning approaches learn from only a single task. In this paper, we propose a novel multi-task deep network to learn generalizable high-level visual representations. Since multi-task learning requires annotations for multiple properties of the same training instance, we look to synthetic images to train our network. To overcome the domain difference between real and synthetic data, we employ an unsupervised feature space domain adaptation method based on adversarial learning. Given an input synthetic RGB image, our network simultaneously predicts its surface normal, depth, and instance contour, while also minimizing the feature space domain differences between real and synthetic data. Through extensive experiments, we demonstrate that our network learns more transferable representations compared to single-task baselines. Our learned representation produces state-of-the-art transfer learning results on PASCAL VOC 2007 classification and 2012 detection.

Motivation & Objective

• To address the high cost and scalability limitations of large-scale annotated datasets like ImageNet for visual representation learning.
• To overcome the challenge of learning general-purpose visual features from synthetic data due to domain shift between synthetic and real images.
• To improve feature transferability by leveraging multiple self-supervised tasks (depth, surface normal, instance contour) on synthetic data.
• To reduce reliance on manual annotations by using free physical property maps from synthetic images as supervisory signals.
• To develop a domain-invariant representation via unsupervised feature-space domain adaptation using adversarial training.

Proposed method

• A multi-task deep network is trained on synthetic RGB images to predict depth, surface normal, and instance contour maps as self-supervised supervisory signals.
• The network uses a shared encoder backbone (e.g., VGG-style) followed by task-specific heads for each prediction task.
• An adversarial domain adaptation module is introduced, where a domain discriminator is trained to distinguish real image features from synthetic image features.
• The generator (feature encoder) is updated to fool the discriminator into classifying real features as synthetic and synthetic features as real, minimizing domain discrepancy in the feature space.
• The overall training objective combines multi-task prediction losses (e.g., L1 for depth, cosine for normals) and adversarial loss for domain alignment.
• The model is pre-trained on 0.5M to 1.5M synthetic images from the SUNCG dataset and fine-tuned on real-world benchmarks like PASCAL VOC and NYUD.

Experimental results

Research questions

• RQ1 Can multi-task self-supervised learning on synthetic data produce more transferable visual representations than single-task baselines?
• RQ2 Does adversarial feature-space domain adaptation effectively reduce the domain gap between synthetic and real images?
• RQ3 How does increasing the scale of synthetic data impact the transferability of learned features?
• RQ4 Can joint prediction of mid-level cues (depth, normal, contour) implicitly encourage learning of high-level semantic features?
• RQ5 How does the proposed method compare to state-of-the-art self-supervised and supervised pre-training approaches on downstream real-world tasks?

Key findings

• The proposed multi-task model achieves 68.0% top-1 accuracy on ImageNet classification and 52.6% mean average precision on PASCAL VOC 2012 detection, outperforming single-task baselines.
• The model achieves 66.7% of pixels with angle error <11.25° on NYUD surface normal estimation, surpassing the self-supervised method of [71] and matching the performance of a supervised method pre-trained on synthetic data.
• Increasing the synthetic dataset size from 0.5M to 1.5M images improves classification and detection accuracy by 0.5–0.6 percentage points, demonstrating data scalability benefits.
• Fooling the discriminator in both directions (real-to-synthetic and synthetic-to-real) slightly degrades performance compared to only fooling it in one direction, indicating suboptimal optimization dynamics.
• The ablation study confirms that multi-task learning, domain adaptation, and data scale all contribute significantly to learning transferable features.
• The model shows strong zero-shot transfer performance, indicating that the self-supervised tasks effectively encourage the learning of high-level semantic representations.