[Paper Review] Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Sandeep Subramanian, Adam Trischler+2 more|arXiv (Cornell University)|Mar 30, 2018

Topic Modeling127 citations

TL;DR

The paper introduces a multi-task learning framework that shares a single recurrent encoder across diverse sentence-level tasks (multilingual NMT, constituency parsing, skip-thought, and natural language inference) to produce general-purpose fixed-length sentence representations that transfer well to new tasks and data-scarce settings.

ABSTRACT

A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.

Motivation & Objective

• Motivate the need for general-purpose sentence representations beyond word embeddings.
• Propose a simple, scalable multi-task framework that combines diverse sentence-level training objectives.
• Show that shared encoding across weakly related tasks improves transfer performance and low-resource learning.

Proposed method

• Use a one-to-many sequence-to-sequence model with a shared bidirectional GRU encoder and task-specific decoders.
• Train on diverse objectives: skip-thought vectors, multilingual neural machine translation (NMT), constituency parsing, and natural language inference (NLI).
• Condition decoders on the encoder representation h_x without using attention, enabling a single fixed-length sentence embedding.
• Interleave tasks during training (uniform task sampling; occasional NLI minibatches) and optimize with Adam.
• Evaluate representations by training a simple linear classifier on transfer tasks without updating the encoder parameters.

Experimental results

Research questions

• RQ1 Does a single shared encoder trained on multiple sentence-level tasks learn more generalizable representations than task-specific or single-objective models?
• RQ2 Do diverse inductive biases from multiple tasks improve transfer performance, especially in low-resource settings?
• RQ3 Which tasks contribute most to capturing syntax, semantics, or other sentence characteristics?
• RQ4 How do fixed-length representations compare to attention-based or task-specific representations on transfer tasks?

Key findings

• Representations learned via multi-task training generalize better across transfer tasks than several prior general-purpose methods.
• Adding more tasks and increasing encoder capacity yields consistent transfer gains on sentiment, entailment, and paraphrase tasks.
• The multi-task model improves low-resource transfer performance, achieving competitive results with only ~6% of labeled data on some tasks.
• Incorporating constituency parsing and multilingual NMT biases enhances syntactic and related linguistic signals in the embeddings.
• The learned word embeddings from the model are competitive with established embedding methods despite being trained from scratch.
• Probing shows that multi-task signals contribute to encoding syntax when parsing and multilingual translation are included, while NLI primarily supports semantic encoding.