[Paper Review] Multitask learning for semantic sequence prediction under varying data conditions.

Hėctor Martínez Alonso, Barbara Plank|arXiv (Cornell University)|Dec 7, 2016

Domain Adaptation and Few-Shot Learning12 citations

TL;DR

This paper investigates multitask learning (MTL) for semantic sequence prediction across varying data conditions, evaluating multiple auxiliary task configurations—including a novel setup—and finding that MTL improves performance in only 1 out of 5 tasks. Success is most likely when auxiliary tasks have compact, uniform label distributions.

ABSTRACT

Multitask learning has been applied successfully to a range of tasks, mostly morphosyntactic. However, little is known on when MTL works and whether there are data characteristics that help to determine the success of MTL. In this paper we evaluate a range of semantic sequence labeling tasks in a MTL setup. We examine different auxiliary task configurations, amongst which a novel setup, and correlate their impact to data-dependent conditions. Our results show that MTL is not always effective, because significant improvements are obtained only for 1 out of 5 tasks. When successful, auxiliary tasks with compact and more uniform label distributions are preferable.

Motivation & Objective

• To evaluate the effectiveness of multitask learning (MTL) in semantic sequence labeling tasks under varying data conditions.
• To identify data-dependent characteristics that predict MTL success or failure.
• To compare different auxiliary task configurations, including a novel setup, for improving semantic sequence prediction.
• To determine whether label distribution uniformity and compactness influence MTL performance.
• To provide empirical insights into when and why MTL works for semantic sequence tasks.

Proposed method

• Employs a multitask learning framework where a shared encoder processes input sequences for multiple semantic sequence labeling tasks.
• Introduces a novel auxiliary task configuration to explore its impact on main task performance.
• Uses shared representation learning across tasks to improve generalization, with task-specific heads for prediction.
• Evaluates performance across five semantic sequence labeling tasks under different data conditions.
• Correlates model performance with data characteristics such as label distribution uniformity and label set compactness.
• Employs standard sequence labeling metrics (e.g., F1-score) to quantify performance gains or declines from MTL.

Experimental results

Research questions

• RQ1 Under which data conditions does multitask learning improve performance in semantic sequence labeling?
• RQ2 How do label distribution characteristics (e.g., uniformity, compactness) affect MTL success?
• RQ3 Does the proposed novel auxiliary task configuration outperform standard configurations?
• RQ4 Is MTL consistently beneficial across different semantic sequence tasks, or are there task-specific limitations?
• RQ5 What data-dependent factors can predict whether MTL will yield significant improvements?

Key findings

• Multitask learning improves performance in only 1 out of 5 evaluated semantic sequence labeling tasks.
• When MTL is effective, auxiliary tasks with compact and more uniform label distributions lead to better performance.
• Tasks with highly skewed or sparse label distributions show little to no improvement from MTL.
• The novel auxiliary task configuration does not consistently outperform standard setups, suggesting design is less critical than data characteristics.
• Data-dependent factors such as label distribution uniformity are strong predictors of MTL success.
• MTL is not a universally effective strategy for semantic sequence prediction, and its benefits are highly conditional on data properties.