[Paper Review] A Review of Deep Transfer Learning and Recent Advancements

Mohammadreza Iman, Khaled Rasheed+1 more|arXiv (Cornell University)|Jan 19, 2022

COVID-19 diagnosis using AI31 citations

TL;DR

The paper surveys deep transfer learning (DTL), outlining its definitions, taxonomy, popular methods, recent applications, and limitations, with discussions on trends and future directions.

ABSTRACT

Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.

Motivation & Objective

• Define and frame the concept of deep transfer learning (DTL) and its motivation.
• Provide a taxonomy of DTL methods, focusing on network/model-based approaches.
• Review recent applied DTL techniques from the past five years across domains.
• Analyze experimental analyses to identify best practices for applying DTL.
• Discuss limitations (e.g., catastrophic forgetting, biased pre-trained models) and potential solutions and trends.

Proposed method

• Survey and synthesize definitions and taxonomies of DTL from existing literature.
• Categorize well-known DTL methods with emphasis on network/model-based approaches.
• Review recent applied DTL techniques from the last five years and their outcomes.
• Summarize experimental analyses to distill best practices for different scenarios.
• Discuss limitations and propose potential solutions and research trends in DTL.

Experimental results

Research questions

• RQ1 What is the formal definition and taxonomy of deep transfer learning (DTL)?
• RQ2 What are the main network/model-based DTL approaches and how do they differ?
• RQ3 How effective are recent DTL techniques in varied applications, especially with limited data?
• RQ4 What are the key limitations of DTL (e.g., catastrophic forgetting, biased pre-trained models) and what solutions/trends exist?

Key findings

• DTL reduces dependence on large labeled datasets and lowers training costs.
• DTL enabled high-accuracy COVID-19 infection detection on chest X-rays with minimal data.
• DTL can enable deployment on edge devices with resource constraints.
• Limitations include catastrophic forgetting and overly biased pre-trained models.
• The paper discusses possible solutions and identifies research trends in DTL.