[Paper Review] Zero-Shot Deep Domain Adaptation

Kuan–Chuan Peng, Ziyan Wu+1 more|arXiv (Cornell University)|Jul 6, 2017

Domain Adaptation and Few-Shot Learning36 references4 citations

TL;DR

This paper proposes Zero-Shot Deep Domain Adaptation (ZDDA), a novel method that enables domain adaptation and sensor fusion without requiring task-relevant target-domain training data. By leveraging task-irrelevant dual-domain pairs (e.g., grayscale and RGB image pairs from Fashion-MNIST), ZDDA trains a joint network to align source-domain representations with target-domain semantics, achieving strong performance on MNIST→MNIST-M and SUN RGB-D scene classification even when target-domain data is absent during training.

ABSTRACT

Domain adaptation is an important tool to transfer knowledge about a task (e.g. classification) learned in a source domain to a second, or target domain. Current approaches assume that task-relevant target-domain data is available during training. We demonstrate how to perform domain adaptation when no such task-relevant target-domain data is available. To tackle this issue, we propose zero-shot deep domain adaptation (ZDDA), which uses privileged information from task-irrelevant dual-domain pairs. ZDDA learns a source-domain representation which is not only tailored for the task of interest but also close to the target-domain representation. Therefore, the source-domain task of interest solution (e.g. a classifier for classification tasks) which is jointly trained with the source-domain representation can be applicable to both the source and target representations. Using the MNIST, Fashion-MNIST, NIST, EMNIST, and SUN RGB-D datasets, we show that ZDDA can perform domain adaptation in classification tasks without access to task-relevant target-domain training data. We also extend ZDDA to perform sensor fusion in the SUN RGB-D scene classification task by simulating task-relevant target-domain representations with task-relevant source-domain data. To the best of our knowledge, ZDDA is the first domain adaptation and sensor fusion method which requires no task-relevant target-domain data. The underlying principle is not particular to computer vision data, but should be extensible to other domains.

Motivation & Objective

• To address the practical limitation that task-relevant target-domain training data is often unavailable due to technical, budgetary, or regulatory constraints.
• To develop a deep learning method that performs domain adaptation without relying on labeled or unlabeled target-domain data.
• To extend the approach to sensor fusion, enabling robust fusion of RGB and depth data without requiring paired training data in both modalities.
• To demonstrate that ZDDA can outperform existing methods even when they have access to target-domain data.
• To validate the method across diverse datasets including MNIST, Fashion-MNIST, NIST, EMNIST, and SUN RGB-D.

Proposed method

• ZDDA uses task-irrelevant dual-domain pairs (e.g., grayscale and RGB images from Fashion-MNIST) to simulate target-domain representations during training.
• A joint deep neural network is trained with supervision from the task-relevant source domain and alignment loss between source and simulated target representations.
• The method employs a shared encoder to extract features from source-domain data and a domain-specific head for the task of interest (e.g., classification).
• A domain-invariant representation is learned via a contrastive or adversarial loss that encourages feature alignment between source and simulated target domains.
• For sensor fusion, ZDDA trains a joint classifier on source-domain data (e.g., depth) and simulated target-domain data (e.g., RGB), enabling fusion without paired target-domain training data.
• The framework is extended to handle noisy test data by training on noisy data during step 3, improving robustness to real-world sensor degradation.

Experimental results

Research questions

• RQ1 Can domain adaptation be achieved without any access to task-relevant target-domain training data?
• RQ2 Can task-irrelevant dual-domain pairs be used to effectively simulate target-domain representations for domain adaptation?
• RQ3 Does ZDDA outperform existing domain adaptation methods that require target-domain data?
• RQ4 Can ZDDA be applied to sensor fusion tasks without requiring paired training data in both modalities?
• RQ5 How robust is ZDDA to noisy input in either source or target domains during inference?

Key findings

• ZDDA achieves 59.74% accuracy on MNIST-M classification without using any MNIST-M training data, outperforming several state-of-the-art DA methods that do use such data.
• On the SUN RGB-D dataset, ZDDA 3 achieves 63.16% accuracy in 10-scene classification using only depth data and simulated RGB data, outperforming naive fusion under noise.
• ZDDA demonstrates consistent performance across 5-fold and 10-fold cross-validation, with stable results across different data splits and class selections.
• ZDDA 3 shows superior robustness to noisy testing data: it outperforms naive fusion by up to 15% in accuracy under high noise levels in both RGB and depth modalities.
• The method maintains strong performance when trained on noisy data (e.g., black rectangles), and generalizes to other noise models, indicating strong robustness.
• ZDDA achieves state-of-the-art performance in zero-shot domain adaptation and sensor fusion, setting a new benchmark for data-scarce scenarios.