[Paper Review] A Comprehensive Survey on Transfer Learning

Fuzhen Zhuang, Zhiyuan Qi+6 more|arXiv (Cornell University)|Nov 7, 2019

Domain Adaptation and Few-Shot Learning208 references228 citations

TL;DR

This survey systematically reviews over forty transfer learning approaches, focusing on homogeneous transfer learning from data and model perspectives, and compares more than twenty models across three datasets to guide practice.

ABSTRACT

Transfer learning aims at improving the performance of target learners on target domains by transferring the knowledge contained in different but related source domains. In this way, the dependence on a large number of target domain data can be reduced for constructing target learners. Due to the wide application prospects, transfer learning has become a popular and promising area in machine learning. Although there are already some valuable and impressive surveys on transfer learning, these surveys introduce approaches in a relatively isolated way and lack the recent advances in transfer learning. Due to the rapid expansion of the transfer learning area, it is both necessary and challenging to comprehensively review the relevant studies. This survey attempts to connect and systematize the existing transfer learning researches, as well as to summarize and interpret the mechanisms and the strategies of transfer learning in a comprehensive way, which may help readers have a better understanding of the current research status and ideas. Unlike previous surveys, this survey paper reviews more than forty representative transfer learning approaches, especially homogeneous transfer learning approaches, from the perspectives of data and model. The applications of transfer learning are also briefly introduced. In order to show the performance of different transfer learning models, over twenty representative transfer learning models are used for experiments. The models are performed on three different datasets, i.e., Amazon Reviews, Reuters-21578, and Office-31. And the experimental results demonstrate the importance of selecting appropriate transfer learning models for different applications in practice.

Motivation & Objective

• Provide a unified, comprehensive view of transfer learning and its mechanisms.
• Systematically categorize existing transfer learning approaches from data and model perspectives.
• Summarize strategies for reducing cross-domain distribution differences and preserving data properties.
• Demonstrate practical guidance by comparing multiple transfer learning models on standard benchmarks.

Proposed method

• Review more than forty representative transfer learning approaches with emphasis on homogeneous transfer learning.
• Interpret transfer learning via data-based and model-based perspectives.
• Discuss instance weighting, distribution metrics (e.g., MMD), and feature transformation techniques.
• Describe feature augmentation methods (e.g., FAM) and their limitations for heterogeneous tasks.
• Summarize experimental practices by evaluating over twenty models on three datasets (Amazon Reviews, Reuters-21578, Office-31).

Experimental results

Research questions

• RQ1 What are the main data- and model-centered strategies for transferring knowledge across domains?
• RQ2 How do homogeneous and heterogeneous transfer scenarios differ, and what methods address each?
• RQ3 Which transfer learning approaches perform well on standard benchmarks and in what settings?
• RQ4 How can distribution differences between source and target domains be measured and mitigated?

Key findings

• A wide range of transfer learning approaches are categorized into instance-based, feature-based, parameter-based, and relational-based methods.
• Maximum Mean Discrepancy (MMD) and related metrics are commonly used to quantify distribution differences and guide adaptation.
• Feature augmentation and mapping techniques enable common latent representations across domains, with adaptations like FAM and HFA for homogeneous and heterogeneous cases.
• Empirical results illustrate the importance of selecting appropriate transfer learning models for specific applications across standard datasets.
• The survey highlights that negative transfer can occur when domain relatedness is weak or misaligned.