[Paper Review] An efficient framework for learning sentence representations

Lajanugen Logeswaran, Honglak Lee|arXiv (Cornell University)|Mar 7, 2018

Topic Modeling33 references300 citations

TL;DR

This paper introduces QT, a discriminative, efficient framework for learning sentence embeddings from unlabelled text, achieving state-of-the-art unsupervised performance with significant training speedups.

ABSTRACT

In this work we propose a simple and efficient framework for learning sentence representations from unlabelled data. Drawing inspiration from the distributional hypothesis and recent work on learning sentence representations, we reformulate the problem of predicting the context in which a sentence appears as a classification problem. Given a sentence and its context, a classifier distinguishes context sentences from other contrastive sentences based on their vector representations. This allows us to efficiently learn different types of encoding functions, and we show that the model learns high-quality sentence representations. We demonstrate that our sentence representations outperform state-of-the-art unsupervised and supervised representation learning methods on several downstream NLP tasks that involve understanding sentence semantics while achieving an order of magnitude speedup in training time.

Motivation & Objective

• Motivate learning high-quality sentence representations without labelled data.
• Propose a discriminative objective to identify correct context sentences among candidates.
• Enable flexible encoder architectures and efficient training to scale with large corpora.
• Show that learned embeddings improve performance on semantic NLP tasks.
• Provide pre-trained encoders and demonstrate competitive or superior performance to prior methods.

Proposed method

• Represent meaning via sentence embeddings produced by encoders f and g.
• Replace generation with a discriminative objective that selects the correct context sentence from candidates.
• Use a scoring function c(u,v); in practice c is inner product u^T v.
• Train with a cross-entropy loss over candidate contexts (ground-truth more plausible than negatives).
• Concatenate f(s) and g(s) as the sentence representation at test time.
• Experiment with GRU-based RNN encoders, training on BookCorpus and UMBC data, with pre-trained word vectors in MC-QT.

Experimental results

Research questions

• RQ1 Can a discriminative, context-prediction objective produce high-quality sentence embeddings from unlabelled data?
• RQ2 How do different encoder architectures (uni-/bi-directional GRUs, CNN variants, multi-channel inputs) affect performance?
• RQ3 What gains in training efficiency and downstream task accuracy can be achieved relative to generation-based sentence models?
• RQ4 Do pre-trained word vectors and larger corpora further improve representations without sacrificing efficiency?
• RQ5 How do the learned representations compare to supervised and other unsupervised approaches on semantic tasks?

Key findings

• Quartet of QT variants (uni-/bi-/combine-/multi-channel) achieve competitive to state-of-the-art unsupervised results on downstream semantic tasks.
• MultiChannel-QT (MC-QT) with pre-trained GloVe embeddings yields strong performance, surpassing several prior methods.
• The framework trains an order of magnitude faster than prior encoder-decoder based approaches while maintaining or improving accuracy.
• On pooled BookCorpus+UMBC data, strict improvements are observed across many tasks, demonstrating scalability to large corpora.
• Ensembling diverse QT models yields near-supervised performance, occasionally surpassing task-specific supervised baselines in certain tasks.
• The approach provides efficient, flexible encoders suitable for large-scale unsupervised sentence representation learning.