[Paper Review] What you can cram into a single vector: Probing sentence embeddings for linguistic properties

Alexis Conneau, Germán Kruszewski+3 more|arXiv (Cornell University)|May 2, 2018

Topic Modeling41 references276 citations

TL;DR

The paper introduces 10 probing tasks to analyze what linguistic information is encoded in sentence embeddings across various encoders and training methods, and relates probing results to downstream tasks.

ABSTRACT

Although much effort has recently been devoted to training high-quality sentence embeddings, we still have a poor understanding of what they are capturing. "Downstream" tasks, often based on sentence classification, are commonly used to evaluate the quality of sentence representations. The complexity of the tasks makes it however difficult to infer what kind of information is present in the representations. We introduce here 10 probing tasks designed to capture simple linguistic features of sentences, and we use them to study embeddings generated by three different encoders trained in eight distinct ways, uncovering intriguing properties of both encoders and training methods.

Motivation & Objective

• Assess what linguistic properties are retained in fixed-size sentence embeddings across diverse architectures.
• Systematically control probing task design to enable cross-model comparisons.
• Examine how training objectives influence the linguistic information encoded in embeddings.
• Relate probing task performance to downstream classification and inference tasks.

Proposed method

• Define 10 probing tasks targeting surface, syntactic, and semantic properties using single-sentence inputs.
• Evaluate multiple encoders (BiLSTM-last/max, BiLSTM-max, Gated ConvNet) trained with diverse objectives.
• Train encoders on NMT, NLI, AutoEncoder, Seq2Tree, SkipThought, and unsupervised baselines, plus untrained random projections.
• Use a fixed classifier (MLP) on top of pre-learned embeddings to predict probing task labels.
• Compare probing results with downstream SentEval benchmarks to assess correlation.
• Release probing data and tools for public use.

Experimental results

Research questions

• RQ1 What linguistic properties are preserved or recoverable from different sentence embeddings?
• RQ2 How do encoder architecture and training objective shape the information encoded in sentence representations?
• RQ3 Do probing task results correlate with performance on downstream NLP tasks?
• RQ4 Can untrained architectures provide strong priors for sentence representations, and how do they compare to trained models?

Key findings

• Bag-of-Vectors (BoV) surprisingly preserves certain sentence properties, especially surface and some syntactic cues, but lacks word-order sensitivity.
• Encoding architectures trained with the same objective yield embeddings with different linguistic properties, highlighting the architecture prior.
• BiLSTM-max often offers strong probing performance, even before training, indicating intrinsic architectural biases.
• NMT-trained encoders tend to be more linguistically aware on probing tasks than those trained on NLI, though NLI better preserves shallow word-level features for downstream tasks.
• There is a positive correlation between WC (word content) and downstream tasks, while SentLen (sentence length) correlates negatively with many downstream tasks; SOMO and CoordInv probe deeper semantic knowledge.