[Paper Review] Multi-task Sequence to Sequence Learning

Minh-Thang Luong, Quoc V. Le+3 more|arXiv (Cornell University)|Nov 19, 2015

Natural Language Processing Techniques66 citations

TL;DR

This paper proposes a multi-task sequence-to-sequence learning framework that improves machine translation, constituent parsing, and image captioning by jointly training on multiple related tasks. It demonstrates up to a 1.5 BLEU point gain in translation and a new state-of-the-art 93.0 F1 score in parsing using shared encoders and decoders across tasks, while revealing distinct behaviors of autoencoder and skip-thought objectives in multi-task settings.

ABSTRACT

Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the oneto-many setting - where the encoder is shared between several tasks such as machine translation and syntactic parsing, (b) the many-to-one setting - useful when only the decoder can be shared, as in the case of translation and image caption generation, and (c) the many-to-many setting - where multiple encoders and decoders are shared, which is the case with unsupervised objectives and translation. Our results show that training on a small amount of parsing and image caption data can improve the translation quality between English and German by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks. Furthermore, we have established a new state-of-the-art result in constituent parsing with 93.0 F1. Lastly, we reveal interesting properties of the two unsupervised learning objectives, autoencoder and skip-thought, in the MTL context: autoencoder helps less in terms of perplexities but more on BLEU scores compared to skip-thought.

Motivation & Objective

• To explore multi-task learning (MTL) in sequence-to-sequence models across diverse NLP tasks such as translation, parsing, and image captioning.
• To investigate whether small-scale auxiliary tasks (e.g., parsing, image captioning) can improve performance on large-scale sequence-to-sequence tasks like machine translation.
• To evaluate the effectiveness of unsupervised objectives—autoencoders and skip-thought vectors—in enhancing supervised sequence-to-sequence learning.
• To understand the divergent behaviors of autoencoder and skip-thought objectives in multi-task settings, particularly regarding intrinsic (perplexity) and extrinsic (BLEU) metrics.
• To establish new state-of-the-art results in constituent parsing through ensembled multi-task models.

Proposed method

• Proposes three MTL settings: one-to-many (shared encoder across tasks), many-to-one (shared decoder), and many-to-many (shared encoders and decoders for multiple tasks).
• Employs sequence-to-sequence models with attention mechanisms for all tasks, using LSTMs as the underlying RNN units.
• Integrates auxiliary tasks by training on a weighted combination of objectives: e.g., translation + parsing or translation + autoencoder/skip-thought.
• Uses mixing coefficients (e.g., 0.05, 0.1) to balance the loss from the main task (translation) and auxiliary tasks (parsing, autoencoding, skip-thought).
• Applies ensembling of multiple multi-task models trained at different mixing ratios to improve generalization and performance on parsing.
• Evaluates models using standard metrics: BLEU for translation, F1 for parsing, and perplexity for unsupervised objectives.

Experimental results

Research questions

• RQ1 Can training on syntactic parsing and image captioning data improve neural machine translation performance despite the data size disparity?
• RQ2 How do different multi-task learning configurations (one-to-many, many-to-one, many-to-many) affect sequence-to-sequence model performance?
• RQ3 Do unsupervised objectives like autoencoders and skip-thought vectors improve translation quality when jointly trained with supervised tasks?
• RQ4 Why do autoencoder and skip-thought objectives exhibit divergent behaviors in terms of perplexity and BLEU scores in multi-task learning?
• RQ5 Can multi-task learning with small auxiliary datasets lead to state-of-the-art results in constituent parsing?

Key findings

• Training on a small amount of parsing and image caption data improved English-German translation by up to 1.5 BLEU points over strong single-task baselines on the WMT benchmarks.
• An ensemble of six multi-task models achieved a new state-of-the-art F1 score of 93.0 in English constituent parsing, surpassing prior single-task systems.
• The autoencoder objective improved BLEU scores more than skip-thought but led to worse perplexity, indicating a trade-off between intrinsic and extrinsic performance.
• Skip-thought vectors improved perplexity consistently with more training data but degraded BLEU scores as mixing ratio increased, suggesting incompatibility with the translation task’s objective.
• The many-to-many MTL setting with autoencoders achieved a 0.5 BLEU point improvement at a mixing ratio of 0.05, but performance dropped with higher ratios.
• The results suggest that unsupervised objectives should be compatible with the primary task—autoencoders are compatible as they resemble monolingual translation, while skip-thought is not.