[Paper Review] Multi-Task Learning for Sequence Tagging: An Empirical Study

Soravit Changpinyo, Hexiang Hu+1 more|arXiv (Cornell University)|Aug 13, 2018

Natural Language Processing Techniques60 references57 citations

TL;DR

The paper empirically compares three MTL approaches for 11 sequence tagging tasks, showing that All or Oracle MTL can outperform STL in roughly half the cases and revealing task relationships and embeddings that cluster tasks semantically.

ABSTRACT

We study three general multi-task learning (MTL) approaches on 11 sequence tagging tasks. Our extensive empirical results show that in about 50% of the cases, jointly learning all 11 tasks improves upon either independent or pairwise learning of the tasks. We also show that pairwise MTL can inform us what tasks can benefit others or what tasks can be benefited if they are learned jointly. In particular, we identify tasks that can always benefit others as well as tasks that can always be harmed by others. Interestingly, one of our MTL approaches yields embeddings of the tasks that reveal the natural clustering of semantic and syntactic tasks. Our inquiries have opened the doors to further utilization of MTL in NLP.

Motivation & Objective

• Investigate whether joint learning of many sequence tagging tasks provides benefits beyond pairwise learning or independent learning.
• Characterize inter-task relationships to identify which tasks help or harm others in MTL.
• Explore how different MTL architectures affect sharing and performance across diverse tag sets.
• Assess whether an Oracle set of beneficial tasks can improve performance over All MTL and STL.

Proposed method

• Describe and implement three MTL frameworks: Multi-Dec (shared encoder, multiple decoders), TE ⊕ Dec (shared encoder and decoder with task embeddings fed to the decoder), TE ⊕ Enc (task embeddings prepended to input to encoder).
• Use a common encoder/decoder setup based on biRNNs with GRU units and CRF classifiers across 11 sequence tagging tasks.
• Train with a balanced mini-batch strategy across tasks and evaluate with span-based micro-averaged F1 scores; compare STL, Pairwise MTL, All MTL, Oracle MTL, and All-but-one MTL settings.
• Construct Oracle sets from pairwise results by selecting tasks that improve the testing task over STL, and analyze how Oracle compares to All MTL.
• Provide extensive cross-task analysis including pairwise benefit/harm graphs and task contribution tables.
• Experiment setup uses datasets such as UD v1.4, CoNLL-2000/2003, Streusle, SemCor, and others, with label spaces and entropy listed in the paper.

Experimental results

Research questions

• RQ1 Can jointly learning more than two tasks improve performance for sequence tagging compared to STL or simple pairwise MTL?
• RQ2 What are the pairwise beneficial or harmful relationships between tasks, and how do these relations scale when learning all tasks jointly?
• RQ3 Do Oracle-based selections of beneficial tasks consistently outperform All MTL, and under what conditions?
• RQ4 Are there emergent embeddings of tasks that reveal natural clustering of semantic and syntactic tasks?

Key findings

• In about 50% of cases, All or Oracle MTL improves over STL or pairwise learning across 11 tasks.
• Pairwise MTL reveals clear beneficial/harmful relationships; some tasks (e.g., mwe, supsense, semtr, hyp) tend to be beneficial to others, while others (e.g., com, frame, hyp) tend to be harmful.
• All MTL often matches or exceeds Pairwise MTL and, in many cases, is surpassed by Oracle MTL, which leverages identified beneficial tasks.
• Task embeddings used in TE-based models yield embeddings that cluster syntactic versus semantic tasks, suggesting meaningful task representations.
• In several tasks (notably com), All MTL can outperform STL even when no single pairwise task improves STL, indicating beneficial regularization across tasks.
• The predictive power of pairwise MTL for All MTL exists but is imperfect; the direction is generally reliable, while magnitude is not easily predicted.