[Paper Review] An Overview of Multi-Task Learning in Deep Neural Networks

Sebastian Ruder|arXiv (Cornell University)|Jun 15, 2017

Domain Adaptation and Few-Shot Learning45 references2,413 citations

TL;DR

A comprehensive overview of multi-task learning (MTL) in deep neural networks, detailing hard and soft parameter sharing, mechanisms for why MTL works, historical and recent literature, and guidance on choosing auxiliary tasks.

ABSTRACT

Multi-task learning (MTL) has led to successes in many applications of machine learning, from natural language processing and speech recognition to computer vision and drug discovery. This article aims to give a general overview of MTL, particularly in deep neural networks. It introduces the two most common methods for MTL in Deep Learning, gives an overview of the literature, and discusses recent advances. In particular, it seeks to help ML practitioners apply MTL by shedding light on how MTL works and providing guidelines for choosing appropriate auxiliary tasks.

Motivation & Objective

• Motivate multi-task learning as a form of inductive transfer that can improve generalization by leveraging related tasks.
• Summarize the two core deep learning MTL approaches (hard and soft parameter sharing) and their trade-offs.
• Survey the mechanisms and literature behind MTL’s effectiveness and regularization effects.
• Discuss how to select and design auxiliary tasks to benefit a main task in MTL.

Proposed method

• Explain hard parameter sharing where common hidden layers are shared across tasks with task-specific output heads.
• Explain soft parameter sharing where each task has its own model but parameters are regularized to stay similar across tasks.
• Describe regularization mechanisms and how they relate to block-sparse and low-rank representations in multi-task settings.
• Discuss the concept of implicit data augmentation, attention focusing, eavesdropping, and representation bias as why MTL helps.
• Survey non-neural MTL literature to contrast with deep learning approaches.
• Introduce recent deep learning MTL approaches (e.g., Deep Relationship Networks, Cross-stitch Networks, Sluice Networks) and their aims to learn what to share.

Experimental results

Research questions

• RQ1 What are the primary methods used to implement MTL in deep neural networks and their respective advantages?
• RQ2 What mechanisms explain the generalization benefits of MTL in practice?
• RQ3 How have recent deep learning approaches advanced the ability to learn what to share across tasks?
• RQ4 What constitutes effective auxiliary tasks for achieving MTL gains in practice?

Key findings

• Hard parameter sharing remains the standard approach in neural MTL and reduces overfitting relative to task-specific heads.
• Soft parameter sharing uses regularization to keep task models related while preserving task-specificity.
• MTL provides mechanisms such as implicit data augmentation, attention focusing, eavesdropping, representation bias, and regularization that explain improved generalization.
• Recent deep learning MTL work seeks to learn what to share and model task hierarchies, going beyond fixed sharing schemes.
• Auxiliary tasks can be related, adversarial, or hint-based, and their design influences MTL gains across domains like NLP, vision, and speech.