[Paper Review] Label Efficient Learning of Transferable Representations across Domains and Tasks

Zelun Luo, Yuliang Zou+2 more|arXiv (Cornell University)|Nov 30, 2017

Domain Adaptation and Few-Shot LearningComputer Science58 references55 citations

TL;DR

A semi-supervised framework that learns domain-invariant representations and semantic transfer to enable transfer across domains and tasks with limited labels, validated on SVHN→MNIST and ImageNet→UCF-101.

ABSTRACT

We propose a framework that learns a representation transferable across different domains and tasks in a label efficient manner. Our approach battles domain shift with a domain adversarial loss, and generalizes the embedding to novel task using a metric learning-based approach. Our model is simultaneously optimized on labeled source data and unlabeled or sparsely labeled data in the target domain. Our method shows compelling results on novel classes within a new domain even when only a few labeled examples per class are available, outperforming the prevalent fine-tuning approach. In addition, we demonstrate the effectiveness of our framework on the transfer learning task from image object recognition to video action recognition.

Motivation & Objective

• Learn a representation transferable across domains with differing label spaces.
• Reduce target-domain label requirements via semi-supervised and unsupervised signals.
• Align source and target feature distributions with a multi-layer domain adversarial loss.
• Enable semantic transfer when label sets are non-overlapping.
• Demonstrate transfer across vision domains (images to video) and tasks (classification to action recognition).

Proposed method

• Joint domain and semantic transfer with a multi-layer domain discriminator for deeper alignment.
• Domain adversarial objective that aligns source and target embeddings across multiple network layers.
• Semantic transfer objective using entropy minimization over similarities between unlabeled target data and labeled data, with softmax temperature control.
• Cross-category similarity to transfer information when label spaces differ, using both unsupervised and supervised components.
• Metric-based cross-entropy loss on labeled target data to stabilize target-domain learning.
• Total objective combines supervised target loss, domain transfer loss, and semantic transfer loss with tunable weights.

Experimental results

Research questions

• RQ1 Can a shared embedding learned from a labeled source domain be effectively transferred to a sparsely labeled or unlabeled target domain with non-overlapping label spaces?
• RQ2 Does multi-layer domain adversarial training improve cross-domain alignment beyond single-layer approaches?
• RQ3 Can semantic transfer via pairwise similarity and temperature-controlled softmax enable transfer when source and target label spaces do not fully overlap?
• RQ4 How well does the proposed framework transfer between substantially different modalities, such as images to video?
• RQ5 What is the impact of unlabeled target data and sparse target labels on final target performance?

Key findings

• The proposed method outperforms fine-tuning, especially with very few labeled target examples.
• Multi-layer domain adversarial loss yields improved target classification and more stable training than single-layer approaches.
• Semantic transfer via entropy minimization with a temperature parameter enables transfer even with non-overlapping label spaces.
• The approach enables transfer from ImageNet images to UCF-101 video action recognition, showing cross-task transfer capability.
• In unsupervised domain adaptation experiments, the multi-layer approach achieves higher accuracy than competing domain-adversarial methods.