[论文解读] Multi-Task Learning in Natural Language Processing: An Overview

Shijie Chen, Yu Zhang+1 more|arXiv (Cornell University)|Sep 19, 2021

Topic Modeling参考文献 145被引用 29

一句话总结

本论文综述了多任务学习（MTL）在自然语言处理中的应用，分类架构、优化技术与应用，并讨论数据集及未来方向。

ABSTRACT

Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

研究动机与目标

• Motivate the use of MTL to address data scarcity and overfitting in NLP tasks.
• Categorize NLP MTL architectures and explain their design principles.
• Summarize optimization techniques for training MTL models in NLP.
• Review applications of auxiliary and joint MTL across NLP tasks and datasets.
• Discuss future research directions and benchmarks for MTL in NLP.

提出的方法

• Classify MTL architectures into parallel, hierarchical, modular, and generative adversarial categories.
• Explain hard vs soft parameter sharing and their implications for model design.
• Describe optimization approaches including loss construction, data sampling, and task scheduling.
• Summarize mechanisms for feature fusion, supervision at different levels, and hierarchical interactions.
• Discuss generative-adversarial MTL and semi-supervised/unlabeled data utilization.
• Provide overview of MTL adapters and multi-lingual/multi-domain modular designs.

实验结果

研究问题

• RQ1 What MTL architectures are most effective for NLP tasks and why?
• RQ2 How can optimization strategies (loss weighting, sampling, scheduling) improve MTL training in NLP?
• RQ3 What are the typical applications and datasets used to evaluate MTL in NLP?
• RQ4 How do modular and adapter-based approaches enable scalable multi-task learning with large pre-trained models?
• RQ5 What future directions and challenges are identified for MTL in NLP?

主要发现

• MTL architectures in NLP are categorized into four classes: parallel, hierarchical, modular, and generative adversarial.
• Hard parameter sharing is common; soft sharing offers flexible task-specific constraints.
• Optimization strategies include combining task losses with weighting, dynamic sampling, and task scheduling to balance learning.
• Hierarchical and modular designs enable effective sharing and task-specific specialization, including adapters for large pre-trained models.
• Generative adversarial setups can encourage task-invariant shared representations and leverage unlabeled data.
• MTL improves data efficiency and robustness, particularly when auxiliary/self-supervised tasks are incorporated.