[Paper Review] Universal Sentence Encoder

Daniel Cer, Yinfei Yang+11 more|arXiv (Cornell University)|Mar 29, 2018

Topic Modeling18 references1,292 citations

TL;DR

The paper introduces two sentence encoding models (Transformer-based and Deep Averaging Network) that produce transferable sentence embeddings for diverse NLP tasks, analyzes data, resources, bias, and shows strong low-resource transfer performance.

ABSTRACT

We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.

Motivation & Objective

• Address data scarcity in NLP by providing high-quality sentence embeddings that transfer well to downstream tasks.
• Compare two encoding architectures (Transformer vs DAN) in terms of accuracy, speed, and resource usage.
• Evaluate transfer learning performance across multiple NLP tasks with varying amounts of task-specific labeled data.
• Investigate potential biases in sentence embeddings using WEAT and compare with word-level baselines.

Proposed method

• Two sentence encoders are proposed: a Transformer-based encoder and a Deep Averaging Network (DAN) encoder.
• Both models are trained using multi-task objectives and augmented with supervised data from SNLI.
• Embeddings are 512-dimensional vectors fed into downstream task models; evaluation includes semantic textual similarity and transfer tasks.
• Transfer task predictions use either task-specific neural networks or direct similarity between sentence embeddings.
• Similarity between sentence embeddings is measured using Angular distance as defined in the paper.

Experimental results

Research questions

• RQ1 How do Transformer-based and DAN-based universal sentence encoders compare in transfer task performance?
• RQ2 What are the trade-offs between model complexity, compute resources, and data requirements for achieving strong transfer performance?
• RQ3 Do sentence-level embeddings outperform word-level transfer baselines across various NLP tasks?
• RQ4 How does the addition of sentence-level and word-level transfer affect performance when task data is scarce?
• RQ5 What biases are detectable in the universal sentence encoder via WEAT analyses, and how do they compare to word embedding biases?

Key findings

• Transformer-based USE generally matches or surpasses DAN-based USE in transfer tasks, with higher accuracy at increased resource cost.
• DAN offers faster inference and lower memory usage with competitive performance on several tasks.
• Sentence-level transfer learning typically outperforms word-level transfer alone across most tasks, and combining both yields the best overall results.
• With limited target task data, USE_T (Transformer) achieves strong performance, nearing or matching models trained on much larger data.
• WEAT analysis shows the DAN variant exhibits biases similar to but generally weaker than GloVe on several metrics; results depend on training data composition.
• Pre-trained encoders are released for TF Hub usage and research.