[论文解读] Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation

Haoxiang Wang, Han Zhao+1 more|arXiv (Cornell University)|Jun 16, 2021

Domain Adaptation and Few-Shot Learning参考文献 63被引用 19

一句话总结

本文建立了多任务学习（MTL）与基于梯度的元学习（GBML）之间的理论与实证联系，表明二者具有相同的优化公式，并且在过参数化的神经网络中，对未见任务的预测结果几乎完全一致。主要贡献是一项一阶MTL方法，其在少样本分类任务上达到了最先进性能，同时训练速度比二阶GBML方法（如MetaOptNet）快一个数量级。

ABSTRACT

Multi-task learning (MTL) aims to improve the generalization of several related tasks by learning them jointly. As a comparison, in addition to the joint training scheme, modern meta-learning allows unseen tasks with limited labels during the test phase, in the hope of fast adaptation over them. Despite the subtle difference between MTL and meta-learning in the problem formulation, both learning paradigms share the same insight that the shared structure between existing training tasks could lead to better generalization and adaptation. In this paper, we take one important step further to understand the close connection between these two learning paradigms, through both theoretical analysis and empirical investigation. Theoretically, we first demonstrate that MTL shares the same optimization formulation with a class of gradient-based meta-learning (GBML) algorithms. We then prove that for over-parameterized neural networks with sufficient depth, the learned predictive functions of MTL and GBML are close. In particular, this result implies that the predictions given by these two models are similar over the same unseen task. Empirically, we corroborate our theoretical findings by showing that, with proper implementation, MTL is competitive against state-of-the-art GBML algorithms on a set of few-shot image classification benchmarks. Since existing GBML algorithms often involve costly second-order bi-level optimization, our first-order MTL method is an order of magnitude faster on large-scale datasets such as mini-ImageNet. We believe this work could help bridge the gap between these two learning paradigms, and provide a computationally efficient alternative to GBML that also supports fast task adaptation.

研究动机与目标

• 理解多任务学习（MTL）与元学习之间的理论与实践关系。
• 解决MTL中训练效率与元学习中快速适应能力之间的权衡。
• 开发一种计算高效的GBML替代方法，支持对未见任务的快速适应。
• 通过实证验证，MTL在少样本基准测试中可达到与最先进GBML算法相当的性能。

提出的方法

• 理论分析表明，MTL与一类GBML算法共享相同的双层优化公式。
• 对于具有足够深度的过参数化神经网络，证明了MTL与GBML的预测函数在函数空间中接近。
• 理论分析表明，MTL与GBML预测之间的差距与网络深度成反比。
• 通过合成数据进行实证验证，表明随着网络深度增加，MTL与GBML的预测相似性逐步提升。
• 开发并评估了一种一阶MTL实现方法，应用于mini-ImageNet及其他少样本基准。
• 该方法采用共享表示的联合训练，并在元测试阶段对特征进行线性分类器微调。

实验结果

研究问题

• RQ1 MTL与GBML是否具有相同的底层优化公式？
• RQ2 对于过参数化的网络，MTL能否在未见任务上实现与GBML相近的预测性能？
• RQ3 MTL与GBML预测之间的差距是否与网络深度成反比？
• RQ4 一阶MTL方法能否在准确率与训练速度上均优于二阶GBML方法？
• RQ5 MTL是否在保持训练效率的同时，支持对未见少样本任务的快速适应？

主要发现

• MTL与GBML共享相同的优化公式，其中GBML可视为MTL联合训练的一种正则化变体。
• 对于过参数化的网络，MTL与GBML的预测函数接近，且差距随网络深度增加而减小。
• 在合成数据上，MTL与GBML的预测相似性随网络深度增加而提升，验证了理论分析。
• 在mini-ImageNet上，所提出的MTL方法在少样本分类准确率上与MetaOptNet（最先进GBML方法）相当或更优。
• 由于采用一阶优化，该MTL方法的训练速度比MetaOptNet快一个数量级，且未损失性能。
• 通过理论扩展，该方法在残差ReLU网络及含LayerNorm的网络上也具有泛化能力。