[论文解读] Towards Understanding the Transferability of Deep Representations

Hong Liu, Mingsheng Long+2 more|arXiv (Cornell University)|Sep 26, 2019

Advanced Neural Network Applications参考文献 30被引用 14

一句话总结

本文通过分析预训练表征如何提升泛化能力和优化性能，探究了深度神经网络中迁移性的机制。研究发现，迁移模型收敛至更平坦的极小值，损失曲面更平滑且Lipschitz常数更优，这是由于梯度抑制所致；迁移性在中期训练阶段达到峰值后开始下降，相关发现获得理论支持。

ABSTRACT

Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can significantly boost generalization and accelerate training if fine-tuned to a smaller target dataset. Despite its pervasiveness, few effort has been devoted to uncovering the reason of transferability in deep feature representations. This paper tries to understand transferability from the perspectives of improved generalization, optimization and the feasibility of transferability. We demonstrate that 1) Transferred models tend to find flatter minima, since their weight matrices stay close to the original flat region of pretrained parameters when transferred to a similar target dataset; 2) Transferred representations make the loss landscape more favorable with improved Lipschitzness, which accelerates and stabilizes training substantially. The improvement largely attributes to the fact that the principal component of gradient is suppressed in the pretrained parameters, thus stabilizing the magnitude of gradient in back-propagation. 3) The feasibility of transferability is related to the similarity of both input and label. And a surprising discovery is that the feasibility is also impacted by the training stages in that the transferability first increases during training, and then declines. We further provide a theoretical analysis to verify our observations.

研究动机与目标

• 理解深度表征在不同数据集和任务间表现出强迁移性的根本原因。
• 探究迁移学习如何提升模型的泛化能力和优化稳定性。
• 分析梯度动态与损失曲面特性在迁移性中的作用。
• 探索预训练时长与迁移性能之间的关系。
• 为迁移学习中观察到的实证现象提供理论基础。

提出的方法

• 利用Lipschitz常数分析损失曲面几何，量化迁移后损失曲面平滑性的提升。
• 通过追踪微调过程中权重与预训练参数的距离，实证测量极小值的平坦度。
• 量化迁移表征中梯度幅值的抑制，特别是梯度主成分的抑制。
• 评估源数据集与目标数据集在输入和标签相似性不同时的迁移性能。
• 通过消融实验分析不同预训练轮次，评估迁移性随时间的变化。
• 基于两层全连接网络构建理论分析，以解释实证观察结果。

实验结果

研究问题

• RQ1 为何迁移后的深度特征能提升微调模型的泛化能力？
• RQ2 迁移学习如何改善目标任务的优化曲面？
• RQ3 梯度动态在稳定和加速迁移训练中的作用是什么？
• RQ4 输入和标签分布的相似性如何影响迁移性？
• RQ5 预训练轮次如何影响特征的迁移性？

主要发现

• 由于微调权重保持在原始预训练参数的平坦区域附近，迁移模型收敛至更平坦的极小值。
• 可迁移表征提升了损失曲面的Lipschitz常数，从而带来更稳定和更快的训练。
• 梯度主成分中的梯度抑制是稳定迁移模型训练动态的关键原因。
• 迁移性并非在预训练完全收敛时达到峰值，而是在中期训练阶段达到最大，之后开始下降。
• 源任务与目标任务之间输入和标签分布的相似性对高迁移性至关重要。
• 基于两层网络的理论分析验证了实证发现，显示出一致的收敛边界与泛化行为。