[论文解读] Transferability in Deep Learning: A Survey

Junguang Jiang, Yang Shu+2 more|arXiv (Cornell University)|Jan 15, 2022

Domain Adaptation and Few-Shot Learning被引用 80

一句话总结

本综述将预训练、适应与评估与深度学习中的可迁移性联系起来，并介绍 TLlib 用于可迁移性方法的公平基准测试。

ABSTRACT

The success of deep learning algorithms generally depends on large-scale data, while humans appear to have inherent ability of knowledge transfer, by recognizing and applying relevant knowledge from previous learning experiences when encountering and solving unseen tasks. Such an ability to acquire and reuse knowledge is known as transferability in deep learning. It has formed the long-term quest towards making deep learning as data-efficient as human learning, and has been motivating fruitful design of more powerful deep learning algorithms. We present this survey to connect different isolated areas in deep learning with their relation to transferability, and to provide a unified and complete view to investigating transferability through the whole lifecycle of deep learning. The survey elaborates the fundamental goals and challenges in parallel with the core principles and methods, covering recent cornerstones in deep architectures, pre-training, task adaptation and domain adaptation. This highlights unanswered questions on the appropriate objectives for learning transferable knowledge and for adapting the knowledge to new tasks and domains, avoiding catastrophic forgetting and negative transfer. Finally, we implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.

研究动机与目标

• 定义可迁移性及其在数据高效深度学习中的作用。
• 提供预训练与适应生命周期的统一视图。
• 回顾预训练（有监督和无监督）和适应（任务与域）的核心方法。
• 突出如灾难性遗忘和负迁移等挑战，并提出用于公平评估的基准测试和库。

提出的方法

• 回顾影响可迁移表征的模型架构和归纳偏置。
• 考察有监督预训练，包括数据质量和领域相似性效应。
• 考察无监督预训练，包括生成式和对比学习。
• 讨论元学习和因果学习作为提高可迁移性的策略。
• 提出评估基准和开源 TLlib，以实现公平方法比较。

实验结果

研究问题

• RQ1 哪些因素决定通用型与任务/域特定的可迁移性？
• RQ2 如何设计预训练和适应以在跨任务与跨域中最大化可迁移性？
• RQ3 哪些原则确保对可迁移性方法的公平、可重复评估？
• RQ4 持续学习、域泛化及相关设置如何为可迁移性研究提供信息？

主要发现

• 预训练质量和模型架构（如深度、基于 Transformer 的设计）对下游的可迁移性有强烈影响。
• 有监督和无监督预训练提供可迁移表征的互补路径，数据规模和任务设计起关键作用。
• 元学习和因果学习提供在不同环境中更快或更鲁棒的适应与泛化的策略。
• 领域自适应理论支撑用于弥合分布变动的实用算法。
• 一个开源的 TLlib 库使预训练和适应方法的公平、可重复基准测试成为可能。