[论文解读] A Review of Deep Transfer Learning and Recent Advancements

Mohammadreza Iman, Khaled Rasheed+1 more|arXiv (Cornell University)|Jan 19, 2022

COVID-19 diagnosis using AI被引用 31

一句话总结

本文综述深度迁移学习（DTL），概述其定义、分类、流行方法、最新应用和局限性，并讨论趋势与未来方向。

ABSTRACT

Deep learning has been the answer to many machine learning problems during the past two decades. However, it comes with two major constraints: dependency on extensive labeled data and training costs. Transfer learning in deep learning, known as Deep Transfer Learning (DTL), attempts to reduce such dependency and costs by reusing an obtained knowledge from a source data/task in training on a target data/task. Most applied DTL techniques are network/model-based approaches. These methods reduce the dependency of deep learning models on extensive training data and drastically decrease training costs. As a result, researchers detected Covid-19 infection on chest X-Rays with high accuracy at the beginning of the pandemic with minimal data using DTL techniques. Also, the training cost reduction makes DTL viable on edge devices with limited resources. Like any new advancement, DTL methods have their own limitations, and a successful transfer depends on some adjustments for different scenarios. In this paper, we review the definition and taxonomy of deep transfer learning and well-known methods. Then we investigate the DTL approaches by reviewing recent applied DTL techniques in the past five years. Further, we review some experimental analyses of DTLs to learn the best practice for applying DTL in different scenarios. Moreover, the limitations of DTLs (catastrophic forgetting dilemma and overly biased pre-trained models) are discussed, along with possible solutions and research trends.

研究动机与目标

• 定义并框定深度迁移学习（DTL）的概念及其动机。
• 提供DTL方法的分类法，重点放在基于网络/模型的方法上。
• 回顾过去五年跨领域的最新应用性DTL技术。
• 分析实验研究以识别应用DTL的最佳实践。
• 讨论局限性（例如灾难性遗忘、偏向的预训练模型）以及潜在的解决方案和趋势。

提出的方法

• 对现有文献中的DTL定义和分类进行调查与综合。
• 对知名DTL方法进行分类，重点放在基于网络/模型的方法上。
• 回顾过去五年中的最新应用性DTL技术及其结果。
• 总结实验分析以提炼不同情景下的最佳实践。
• 讨论局限性并提出DTL的潜在解决方案和研究趋势。

实验结果

研究问题

• RQ1 深度迁移学习（DTL）的正式定义和分类是什么？
• RQ2 主要的基于网络/模型的DTL方法有哪些，它们有何不同？
• RQ3 在各种应用中的近期DTL技术有多有效，尤其是在数据有限的情况下？
• RQ4 DTL的关键局限性有哪些（例如灾难性遗忘、偏向的预训练模型），以及有哪些解决方案/趋势？

主要发现

• DTL减少对大规模带标签数据的依赖并降低训练成本。
• DTL在胸部X线影像上实现了对COVID-19感染的高准确性检测，且数据极少。
• DTL能够在资源受限的边缘设备上部署。
• 局限性包括灾难性遗忘和过度偏向的预训练模型。
• 本文讨论了可能的解决方案并识别了DTL的研究趋势。