[Paper Review] Domain Adaptive Transfer Learning with Specialist Models

Jiquan Ngiam, Daiyi Peng+4 more|arXiv (Cornell University)|Nov 16, 2018

Domain Adaptation and Few-Shot Learning29 references89 citations

TL;DR

The paper analyzes how pre-training data choice affects transfer learning and introduces domain adaptive transfer learning, which uses target-driven importance weights to pre-train on a relevant subset, achieving strong results on fine-grained datasets.

ABSTRACT

Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance. We find that more pre-training data does not always help, and transfer performance depends on a judicious choice of pre-training data. These findings are important given the continued increase in dataset sizes. We further propose domain adaptive transfer learning, a simple and effective pre-training method using importance weights computed based on the target dataset. Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning. Our methods achieve state-of-the-art results on multiple fine-grained classification datasets and are well-suited for use in practice.

Motivation & Objective

• Investigate how the choice of pre-training data impacts transfer learning performance across target datasets.
• Show that larger pre-training data does not always improve transfer, especially for fine-grained tasks.
• Introduce domain adaptive transfer learning that weights source examples by their relevance to the target dataset.
• Demonstrate that importance-weighted pre-training can match or exceed hand-picked, domain-specific subsets.

Proposed method

• Define importance weights as the ratio of target label distribution to source label distribution, P_t(y)/P_s(y).
• Estimate P_t(y) using a classifier trained on the source data and predictions on target data without using target labels.
• Up-weight source examples during pre-training by the calculated importance weights and sample a pre-training dataset accordingly.
• Compare adaptive transfer to hand-picked pre-training subsets and standard baselines using large source datasets (JFT, ImageNet) and standard models (Inception v3, AmoebaNet-B).
• Apply a temperature parameter to softmax when computing P_t(y) to stabilize importance weighting.

Experimental results

Research questions

• RQ1 Does weighting source pre-training data by target-relevant importance improve transfer performance across multiple target datasets?
• RQ2 How does domain similarity between source and target affect transfer outcomes and potential negative transfer?
• RQ3 How does domain adaptive transfer learning compare with hand-picked or other data selection methods on fine-grained tasks?
• RQ4 Do larger models retain the same qualitative trends when using domain-adaptive pre-training?

Key findings

• More pre-training data is not always better; selective subsets that discount irrelevant examples improve transfer.
• Domain adaptive transfer learning yields better or competitive results versus hand-picked label subsets and can require less pre-training data to achieve strong performance.
• Transfer is strongest when the source domain closely matches the target task; mismatches can lead to negative transfer.
• Adaptive transfer improves performance on several fine-grained datasets and is competitive with or better than state-of-the-art hand-selected subsets, even with large models.
• Large models (e.g., AmoebaNet-B) show the same qualitative gains from domain-adaptive pre-training, sometimes narrowing gaps between general and specific subsets.