[Paper Review] Deep Transfer Learning with Joint Adaptation Networks

Mingsheng Long, Zhu Han+2 more|arXiv (Cornell University)|May 21, 2016

Domain Adaptation and Few-Shot LearningComputer Science42 references115 citations

TL;DR

JAN learns transferable features by aligning the joint distributions of multi-layer activations across domains using a joint maximum mean discrepancy (JMMD) criterion, with an adversarial variant JAN-A to maximize JMMD during training.

ABSTRACT

Deep networks have been successfully applied to learn transferable features for adapting models from a source domain to a different target domain. In this paper, we present joint adaptation networks (JAN), which learn a transfer network by aligning the joint distributions of multiple domain-specific layers across domains based on a joint maximum mean discrepancy (JMMD) criterion. Adversarial training strategy is adopted to maximize JMMD such that the distributions of the source and target domains are made more distinguishable. Learning can be performed by stochastic gradient descent with the gradients computed by back-propagation in linear-time. Experiments testify that our model yields state of the art results on standard datasets.

Motivation & Objective

• Motivate unsupervised domain adaptation where joint distributions of inputs and labels shift across domains.
• Develop a deep learning framework that aligns joint activations across domain-specific layers.
• Propose JMMD to measure and minimize joint distribution discrepancy in multi-layer features.
• Provide an end-to-end trainable method compatible with SGD for scalable deep transfer learning.

Proposed method

• Introduce Joint Adaptation Networks (JAN) to align joint distributions of activations across domain-specific layers using JMMD.
• Define JMMD as the squared distance between empirical joint embeddings in a tensor product RKHS.
• Provide a linear-time unbiased estimator of JMMD for mini-batch SGD.
• In JAN, minimize source classification loss plus JMMD penalty on domain-specific layers to adapt features and classifiers.
• Propose JAN-A by adding a neural adversary (θ) to maximize JMMD, forming a min-max objective.
• Use progressive scheduling for the adaptation weight λ to stabilize training.

Experimental results

Research questions

• RQ1 Can aligning the joint distributions of multi-layer activations across domains improve unsupervised domain adaptation performance?
• RQ2 How can JMMD be computed and optimized efficiently in deep networks to enable end-to-end training?
• RQ3 Does adversarially maximizing JMMD (JAN-A) provide practical benefits over standard JAN?
• RQ4 Which network layers should be joint-adapted to effectively transfer across domains?

Key findings

• JAN and JAN-A achieve state-of-the-art results on Office-31 and ImageCLEF-DA benchmarks compared to multiple prior methods.
• JAN outperforms several baselines (e.g., DAN, RTN, RevGrad) in average accuracy across transfer tasks on Office-31 with both AlexNet and ResNet backbones.
• JAN and JAN-A demonstrate the benefit of aligning joint distributions of higher-layer activations (fc6,fc7,fc8 for AlexNet; pool5,fc for ResNet).
• An unbiased, linear-time JMMD estimator enables efficient integration with mini-batch SGD for deep networks.
• Adversarial JMMD (JAN-A) provides comparable or slightly improved performance over JAN, with training stability enhancements.