[论文解读] Identification of Negative Transfers in Multitask Learning Using Surrogate Models

Dongyue Li, Huy L. Nguyên+1 more|arXiv (Cornell University)|Mar 25, 2023

Domain Adaptation and Few-Shot Learning被引用 5

一句话总结

本文提出了一种代理建模方法，通过采样源任务子集、在其预计算性能上训练线性回归模型，并利用该模型预测未见子集的性能，从而识别多任务学习中的负迁移。该方法通过相关性得分高效识别有益的源任务，在检测负迁移和提升弱监督基准方面显著优于现有亲和度度量方法。

ABSTRACT

Multitask learning is widely used in practice to train a low-resource target task by augmenting it with multiple related source tasks. Yet, naively combining all the source tasks with a target task does not always improve the prediction performance for the target task due to negative transfers. Thus, a critical problem in multitask learning is identifying subsets of source tasks that would benefit the target task. This problem is computationally challenging since the number of subsets grows exponentially with the number of source tasks; efficient heuristics for subset selection do not always capture the relationship between task subsets and multitask learning performances. In this paper, we introduce an efficient procedure to address this problem via surrogate modeling. In surrogate modeling, we sample (random) subsets of source tasks and precompute their multitask learning performances. Then, we approximate the precomputed performances with a linear regression model that can also predict the multitask performance of unseen task subsets. We show theoretically and empirically that fitting this model only requires sampling linearly many subsets in the number of source tasks. The fitted model provides a relevance score between each source and target task. We use the relevance scores to perform subset selection for multitask learning by thresholding. Through extensive experiments, we show that our approach predicts negative transfers from multiple source tasks to target tasks much more accurately than existing task affinity measures. Additionally, we demonstrate that for several weak supervision datasets, our approach consistently improves upon existing optimization methods for multitask learning.

研究动机与目标

• 为解决多任务学习中的负迁移问题，即合并所有源任务会损害目标任务性能。
• 开发一种高效的方法，以选择能增强目标任务学习的最优源任务子集。
• 降低评估所有指数级增长的源任务子集的计算负担。
• 改进现有启发式子集选择方法，这些方法无法捕捉复杂的任务关系。
• 实现对未见任务子集的多任务学习性能的准确预测。

提出的方法

• 采样源任务的随机子集以评估其多任务学习性能。
• 使用现有的多任务学习框架预计算每个采样子集的多任务学习性能。
• 训练一个线性回归代理模型，以近似任务子集与其性能之间的关系。
• 利用拟合后的模型预测未见任务子集的性能，而无需重新训练。
• 从代理模型的系数中推导出每个源任务与目标任务之间的相关性得分。
• 通过阈值化相关性得分执行子集选择，以排除低贡献或有害的源任务。

实验结果

研究问题

• RQ1 代理建模能否有效预测未见源任务子集的多任务学习性能？
• RQ2 与现有任务亲和度度量相比，所提出方法在检测负迁移方面表现如何？
• RQ3 与标准多任务学习优化方法相比，代理模型方法是否能在弱监督数据集上提升目标任务性能？
• RQ4 该方法在所需采样子集数量方面的计算效率如何？
• RQ5 代理模型的相关性得分在多大程度上能指导多任务学习中的有效子集选择？

主要发现

• 代理模型仅需相对于源任务数量呈线性增长的采样子集数量，显著降低了计算成本。
• 与现有任务亲和度度量相比，该方法能更准确地从多个源任务中预测负迁移。
• 在多个弱监督数据集上，所提出方法在多任务学习中持续优于基线优化方法。
• 从代理模型中推导的相关性得分能有效识别有害或无益的源任务，从而实现更优的子集选择。
• 实证结果表明，基于相关性得分的阈值化处理可提升泛化性能并减少负迁移。
• 理论分析证实，代理建模方法在大规模多任务学习场景下具有可扩展性和统计可靠性。