[Paper Review] Learning Task Grouping and Overlap in Multi-task Learning

Abhishek Kumar, Hal Daumé|arXiv (Cornell University)|Jun 27, 2012

Domain Adaptation and Few-Shot Learning31 references252 citations

TL;DR

This paper proposes a multi-task learning framework that models task parameters as sparse linear combinations of shared basis tasks, enabling flexible task grouping and overlap. By enforcing sparsity in the combination coefficients, the method controls information sharing between tasks, outperforming existing approaches on four benchmark datasets.

ABSTRACT

In the paradigm of multi-task learning, mul- tiple related prediction tasks are learned jointly, sharing information across the tasks. We propose a framework for multi-task learn- ing that enables one to selectively share the information across the tasks. We assume that each task parameter vector is a linear combi- nation of a finite number of underlying basis tasks. The coefficients of the linear combina- tion are sparse in nature and the overlap in the sparsity patterns of two tasks controls the amount of sharing across these. Our model is based on on the assumption that task pa- rameters within a group lie in a low dimen- sional subspace but allows the tasks in differ- ent groups to overlap with each other in one or more bases. Experimental results on four datasets show that our approach outperforms competing methods.

Motivation & Objective

• To address the challenge of selecting which tasks should share information in multi-task learning.
• To model task relationships through shared underlying basis tasks with sparse coefficients.
• To allow overlapping task groups by permitting multiple tasks to share the same basis components.
• To improve generalization by learning task groupings and overlaps automatically from data.
• To outperform existing multi-task learning methods on real-world benchmark datasets.

Proposed method

• Task parameters are modeled as sparse linear combinations of a fixed set of basis tasks.
• Sparsity in the combination coefficients controls the extent and pattern of information sharing between tasks.
• The framework uses a low-dimensional subspace assumption for tasks within the same group.
• Overlapping is allowed by permitting tasks to share one or more basis tasks.
• Optimization is performed via a regularized learning objective that encourages sparsity and low-rank structure.
• The method is trained end-to-end using a convex optimization formulation to learn both basis tasks and their coefficients.

Experimental results

Research questions

• RQ1 How can we automatically learn which tasks should share information in multi-task learning?
• RQ2 To what extent can task grouping and overlap improve generalization in multi-task learning?
• RQ3 Can a sparse linear combination of basis tasks effectively model complex task relationships?
• RQ4 How does the proposed method compare to existing multi-task learning baselines in terms of performance?
• RQ5 What is the impact of overlapping task groups on predictive accuracy?

Key findings

• The proposed method outperforms competing multi-task learning approaches on four real-world datasets.
• The model effectively learns task groupings and overlaps without requiring prior knowledge of task relationships.
• Sparsity in the coefficient vectors leads to meaningful and interpretable task groupings.
• The framework achieves better generalization by selectively sharing information across related tasks.
• Empirical results demonstrate consistent performance gains across diverse learning tasks.
• The method is robust to varying degrees of task overlap and group structure.