[Paper Review] Partial Transfer Learning with Selective Adversarial Networks

Zhangjie Cao, Mingsheng Long+2 more|arXiv (Cornell University)|Jul 25, 2017

Domain Adaptation and Few-Shot Learning28 references63 citations

TL;DR

SAN selectively transfers knowledge by aligning distributions only in the shared label space and filtering out outlier source classes, achieving state-of-the-art partial transfer results.

ABSTRACT

Adversarial learning has been successfully embedded into deep networks to learn transferable features, which reduce distribution discrepancy between the source and target domains. Existing domain adversarial networks assume fully shared label space across domains. In the presence of big data, there is strong motivation of transferring both classification and representation models from existing big domains to unknown small domains. This paper introduces partial transfer learning, which relaxes the shared label space assumption to that the target label space is only a subspace of the source label space. Previous methods typically match the whole source domain to the target domain, which are prone to negative transfer for the partial transfer problem. We present Selective Adversarial Network (SAN), which simultaneously circumvents negative transfer by selecting out the outlier source classes and promotes positive transfer by maximally matching the data distributions in the shared label space. Experiments demonstrate that our models exceed state-of-the-art results for partial transfer learning tasks on several benchmark datasets.

Motivation & Objective

• Motivate partial transfer learning where target labels are a subset of source labels.
• Develop a deep framework that avoids negative transfer from outlier source classes.
• Promote positive transfer by distribution matching in the shared label space.
• Enable end-to-end training with an entropy minimization component.
• Demonstrate state-of-the-art performance on benchmark domain adaptation datasets.

Proposed method

• Introduce SAN, extending domain adversarial networks to handle C_t ⊂ C_s by using class-wise domain discriminators.
• Compute a probability-weighted loss over |C_s| domain discriminators based on G_y(x) to assign target instances to relevant discriminators.
• Incorporate a class-level weighting that down-weights discriminators corresponding to outlier source classes C_s \\ C_t.
• Apply entropy minimization on the target unlabeled data to sharpen G_y predictions and improve Y_hat for transfer.
• Provide an end-to-end training objective combining supervised source loss, entropy loss on target, and selective adversarial loss (Equation 6 and 7).
• Evaluate SAN against baselines on Office-31, Caltech-Office, and ImageNet-Caltech with six transfer tasks and ablations SAN-selective and SAN-entropy.

Experimental results

Research questions

• RQ1 Can partial transfer learning be achieved when the target label space is a subset of the source label space?
• RQ2 Does selective adversarial adaptation reduce negative transfer from outlier source classes while promoting positive transfer in the shared space?
• RQ3 Does entropy minimization help refine the label predictor to improve selective transfer?
• RQ4 How does SAN compare to standard adversarial/domain adaptation methods on partial transfer tasks?

Key findings

• SAN outperforms state-of-the-art methods on all six Office-31 transfer tasks.
• SAN achieves substantial gains on tasks with small source or small target spaces and also on large-source-to-small-target settings.
• Ablations show SAN-selective underperforms SAN and SAN-entropy, confirming the benefit of both selective adversarial matching and entropy minimization.
• Entropy minimization enhances performance, especially when class count disparity is large (e.g., 1000→84).
• t-SNE visualizations illustrate sharper separation and correct alignment of target data to relevant source classes under SAN.
• Overall, SAN demonstrates stable convergence and superior handling of negative transfer compared to RevGrad and DAN/RTN baselines.