[Paper Review] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Alexis Conneau, Douwe Kiela+3 more|arXiv (Cornell University)|May 5, 2017

Topic Modeling48 references2,056 citations

TL;DR

The authors show that universal sentence embeddings trained in a supervised manner on SNLI can surpass unsupervised methods (like SkipThought) on a broad set of transfer tasks, with a BiLSTM maximum-pooling encoder providing state-of-the-art results. Training on SNLI enables faster, high-quality transferable sentence representations.

ABSTRACT

Many modern NLP systems rely on word embeddings, previously trained in an unsupervised manner on large corpora, as base features. Efforts to obtain embeddings for larger chunks of text, such as sentences, have however not been so successful. Several attempts at learning unsupervised representations of sentences have not reached satisfactory enough performance to be widely adopted. In this paper, we show how universal sentence representations trained using the supervised data of the Stanford Natural Language Inference datasets can consistently outperform unsupervised methods like SkipThought vectors on a wide range of transfer tasks. Much like how computer vision uses ImageNet to obtain features, which can then be transferred to other tasks, our work tends to indicate the suitability of natural language inference for transfer learning to other NLP tasks. Our encoder is publicly available.

Motivation & Objective

• Investigate how supervised data can be used to learn general-purpose sentence encoders.
• Compare multiple sentence-encoder architectures to identify which best transfers across tasks.
• Demonstrate that training on natural language inference (SNLI) yields stronger transfer performance than other supervised or unsupervised approaches.
• Assess the efficiency and practicality of training and deploying the proposed sentence representations.

Proposed method

• Train sentence encoders on the SNLI natural language inference dataset using a shared encoder to produce representations for premises and hypotheses.
• Apply three matching methods on the encoded sentence pair (concatenation, element-wise product, absolute difference) before a softmax classifier for entailment/contradiction/neutral.
• Evaluate a variety of encoder architectures (LSTM/GRU/BiLSTM with mean or max pooling, self-attentive networks, hierarchical convnets) to assess transferability of embeddings.
• Fix word embeddings (GloVe) and train classifiers on top using logistic regression for transfer evaluation.
• Use SentEval to automate transfer evaluation across 12 downstream tasks, including binary and multi-class classification, entailment, semantic relatedness, and image-caption tasks.

Experimental results

Research questions

• RQ1 Can supervised sentence encoders trained on SNLI provide universal representations that transfer to diverse NLP tasks?
• RQ2 Which encoder architecture best preserves transferable semantic information when trained on SNLI?
• RQ3 Does supervised training on SNLI outperform other supervised or unsupervised sentence representation learning approaches in transfer tasks?
• RQ4 How does embedding size affect transfer performance across architectures?
• RQ5 What is the impact of data diversity (e.g., SNLI vs. MultiNLI) on the generalization of sentence embeddings?

Key findings

• BiLSTM with max pooling trained on SNLI yields strong transfer performance across many tasks, outperforming unsupervised methods like SkipThought on several metrics.
• Some architectures that perform best on the SNLI task can overfit to task biases and underperform on transfer tasks, highlighting the need for architectures that generalize well.
• Larger embedding sizes generally improve transfer performance for several models, suggesting richer representations aid downstream tasks.
• The SNLI-trained embeddings outperform other supervised task-trained embeddings (e.g., COCO, dictionary definitions, NMT) in transfer settings, indicating the semantic structure captured by NLI is highly transferable.
• Training on SNLI (and AllNLI with MultiNLI) improves performance on out-domain tasks like SICK-E and SICK-R, demonstrating robustness across genres and domains.
• In image-caption retrieval, SNLI-trained BiLSTM-max embeddings achieve competitive results, approaching or surpassing some directly trained baselines when paired with strong visual features.