[Paper Review] A Survey on Multi-Task Learning

Yu Zhang, Qiang Yang|arXiv (Cornell University)|Jul 25, 2017

Domain Adaptation and Few-Shot Learning203 references620 citations

TL;DR

This survey reviews Multi-Task Learning (MTL) from algorithmic modeling, applications, and theory, classifying MTL methods into five categories and discussing hybrids with other paradigms and large-scale settings.

ABSTRACT

Multi-Task Learning (MTL) is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks. In this paper, we give a survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses. For algorithmic modeling, we give a definition of MTL and then classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach and decomposition approach as well as discussing the characteristics of each approach. In order to improve the performance of learning tasks further, MTL can be combined with other learning paradigms including semi-supervised learning, active learning, unsupervised learning, reinforcement learning, multi-view learning and graphical models. When the number of tasks is large or the data dimensionality is high, we review online, parallel and distributed MTL models as well as dimensionality reduction and feature hashing to reveal their computational and storage advantages. Many real-world applications use MTL to boost their performance and we review representative works in this paper. Finally, we present theoretical analyses and discuss several future directions for MTL.

Motivation & Objective

• Define Multi-Task Learning and clarify related concepts (homogeneous vs heterogeneous feature spaces).
• Classify existing MTL algorithms into five categories: feature learning, low-rank, task clustering, task relation learning, and decomposition.
• Discuss combinations of MTL with semi-supervised, active, unsupervised, reinforcement, multi-view learning, and graphical models.
• Review scalability considerations (online, parallel, distributed MTL) and high-dimensional techniques (dimensionality reduction, feature hashing).

Proposed method

• Provide a formal definition of MTL for m tasks with relatedness.
• Classify algorithmic approaches into five categories and describe characteristics of each.
• Detail feature-based and parameter-based methods including subtypes (transformation vs selection; low-rank, clustering, relation learning, decomposition).
• Introduce deep MTL variants with shared representations, adversarial nets, and cross-stitch networks.
• Discuss combinations with other learning paradigms and scalability solutions for high-dimensional or many-task settings.

Experimental results

Research questions

• RQ1 What is the precise definition and scope of Multi-Task Learning across supervised and non-supervised settings?
• RQ2 How can MTL algorithms be categorized and what characterizes each category?
• RQ3 What are effective strategies for sharing knowledge across tasks (feature, instance, parameter sharing) and their trade-offs?
• RQ4 How does MTL interact with other learning paradigms and scale to many tasks or high-dimensional data?
• RQ5 What are the key theoretical analyses and future directions for MTL?

Key findings

• MTL can be organized into five main algorithmic categories: feature learning, low-rank, task clustering, task relation learning, and decomposition.
• Feature learning can be implemented via transformation-based, often with deep models, or feature selection with group sparsity regularizers like l21 and related norms.
• Low-rank and decomposition approaches model inter-task relatedness via shared subspaces or matrix factorizations, enabling scalable parameter sharing.
• Hybrid deep MTL models use shared networks, adversarial objectives, or cross-stitch mechanisms to balance common and task-specific representations.
• The survey also discusses combining MTL with semi-supervised, active, unsupervised, reinforcement, multi-view learning, and graphical models, as well as online/parallel/distributed settings and dimensionality reduction techniques.