[Paper Review] The Expressive Power of Word Embeddings

Yanqing Chen, Bryan Perozzi+2 more|arXiv (Cornell University)|Jan 15, 2013

Topic ModelingComputer Science19 references83 citations

TL;DR

This paper evaluates the expressive power of publicly released word embeddings using context-free classification tasks, demonstrating that embeddings capture nuanced semantics even without sentence structure. It reveals significant variation in quality across embeddings and shows that reducing dimensionality or precision has surprisingly small impact, indicating that high-dimensional continuous representations are robust and efficient for NLP tasks.

ABSTRACT

We seek to better understand the difference in quality of the several publicly released embeddings. We propose several tasks that help to distinguish the characteristics of different embeddings. Our evaluation of sentiment polarity and synonym/antonym relations shows that embeddings are able to capture surprisingly nuanced semantics even in the absence of sentence structure. Moreover, benchmarking the embeddings shows great variance in quality and characteristics of the semantics captured by the tested embeddings. Finally, we show the impact of varying the number of dimensions and the resolution of each dimension on the effective useful features captured by the embedding space. Our contributions highlight the importance of embeddings for NLP tasks and the effect of their quality on the final results.

Motivation & Objective

• To understand what linguistic information is encoded in word embeddings beyond sentence context.
• To benchmark publicly available word embeddings and compare their quality and semantic characteristics.
• To investigate how dimensionality and precision affect the utility of word embeddings in downstream tasks.
• To demonstrate that word pair representations can encode more useful linguistic information than individual words.
• To provide a benchmark suite for evaluating word embeddings using isolated term and pair classification tasks.

Proposed method

• Designing context-free classification tasks (sentiment polarity, noun gender, plurality, synonym/antonym, UK/US spelling) to isolate semantic content from syntactic context.
• Using multiple classifiers (SVM, kernel SVM) on embeddings to evaluate performance on the above tasks.
• Applying bitwise truncation to reduce embedding precision from 32-bit floats to binary values (±1) to test resolution requirements.
• Performing Principal Component Analysis (PCA) to reduce embedding dimensionality and assess impact on task performance.
• Comparing performance across four publicly released embeddings: HLBL, SENNA, Turian’s, and Huang’s.
• Analyzing pair-based representations by combining embeddings of word pairs to improve classification beyond individual word features.

Experimental results

Research questions

• RQ1 How well do word embeddings capture semantics in the absence of sentence structure?
• RQ2 What are the key differences in semantic quality and characteristics across publicly released word embeddings?
• RQ3 How does reducing the number of dimensions or the precision of embeddings affect their performance on linguistic tasks?
• RQ4 Can word pair representations encode more useful linguistic information than individual word embeddings?
• RQ5 What is the minimum effective dimensionality and precision required to preserve useful information in word embeddings?

Key findings

• Word embeddings capture surprisingly nuanced semantics—such as sentiment, gender, plurality, and synonymy—without relying on sentence-level context.
• Significant variation exists in the quality and characteristics of different publicly released embeddings, with no single model outperforming all others across all tasks.
• Reducing embedding precision by truncating 31 bits (resulting in binary ±1 values) degrades performance by no more than 7%, indicating high robustness to precision loss.
• Dimensionality reduction via PCA leads to significant performance drops, suggesting that the full embedding space preserves non-linear relationships critical for task performance.
• Pair-based classification using combined word embeddings outperforms individual word classification, with one example showing a substantial performance gain over single-word features.
• The effective resolution of word embeddings is extremely high—equivalent to at least 2^50 distinct regions—yet even binary representations retain strong utility.