[Paper Review] Asymmetric Tri-training for Unsupervised Domain Adaptation

Kuniaki Saito, Yoshitaka Ushiku+1 more|arXiv (Cornell University)|Feb 27, 2017

Domain Adaptation and Few-Shot Learning33 references441 citations

TL;DR

A novel asymmetric tri-training framework for unsupervised domain adaptation uses two classifiers to pseudo-label target data and a third target-focused network trained on those pseudo-labels, achieving state-of-the-art results on multiple visual and sentiment tasks.

ABSTRACT

Deep-layered models trained on a large number of labeled samples boost the accuracy of many tasks. It is important to apply such models to different domains because collecting many labeled samples in various domains is expensive. In unsupervised domain adaptation, one needs to train a classifier that works well on a target domain when provided with labeled source samples and unlabeled target samples. Although many methods aim to match the distributions of source and target samples, simply matching the distribution cannot ensure accuracy on the target domain. To learn discriminative representations for the target domain, we assume that artificially labeling target samples can result in a good representation. Tri-training leverages three classifiers equally to give pseudo-labels to unlabeled samples, but the method does not assume labeling samples generated from a different domain.In this paper, we propose an asymmetric tri-training method for unsupervised domain adaptation, where we assign pseudo-labels to unlabeled samples and train neural networks as if they are true labels. In our work, we use three networks asymmetrically. By asymmetric, we mean that two networks are used to label unlabeled target samples and one network is trained by the samples to obtain target-discriminative representations. We evaluate our method on digit recognition and sentiment analysis datasets. Our proposed method achieves state-of-the-art performance on the benchmark digit recognition datasets of domain adaptation.

Motivation & Objective

• Motivate the need for discriminative target-domain representations when target labels are unavailable.
• Introduce an asymmetric tri-training architecture to generate pseudo-labels for target samples.
• Learn target-discriminative representations while mitigating the impact of noisy pseudo-labels.

Proposed method

• Shared feature extractor F feeds into three classifiers: F1 and F2 trained on source and pseudo-labeled targets; Ft trained only on pseudo-labeled targets.
• Three-way loss for F1 and F2 includes a weight-constraint term |W1^T W2| to encourage diverse views.
• Pseudo-labeling requires agreement between F1 and F2 and a confidence threshold (0.9–0.95) before adding to the target-labeled set.
• Training alternates between updating all networks with S ∪ Tl and updating Ft with Tl to enhance target-discriminative representations.
• Batch Normalization is added in the last layer to help align source and target feature distributions.
• Theoretical framing adapts Ben-David et al. (2010) bounds to account for pseudo-labeled target data and domain divergence.

Experimental results

Research questions

• RQ1 Can asymmetric tri-training improve target-domain accuracy when target labels are absent but pseudo-labels can be confidently assigned?
• RQ2 How does the inclusion of a target-specific network Ft and shared feature extractor F influence target-discriminative representation learning?
• RQ3 Does incorporating pseudo-labeled target samples reduce domain divergence and improve target performance beyond distribution-matching approaches?

Key findings

• Achieved state-of-the-art results on visual domain adaptation benchmarks, notably MNIST to SVHN with more than 10% gains over competitors.
• Outperformed baselines across MNIST→MNIST-M, SVHN→MNIST, SYN DIGITS→SVHN, and SYN SIGNS→GTSRB tasks.
• In Amazon Reviews sentiment transfer, outperformed VFAE and DANN in majority of settings; BN and gradient-sharing analyses highlighted effectiveness under various configurations.
• t-SNE visualizations show target samples becoming more discriminative after adaptation.
• The method reduces the empirical A-distance compared to source-only CNNs, while still delivering superior target accuracy compared to stronger domain-adaptation baselines.
• The three networks F1, F2, Ft tend to converge in performance, indicating shared target-discriminative representations.