[Paper Review] Frequency-based Distortions in Contextualized Word Embeddings

Kaitlyn Zhou, Kawin Ethayarajh+1 more|arXiv (Cornell University)|Apr 17, 2021

Topic Modeling18 references8 citations

TL;DR

This paper investigates how word frequency in pre-training data distorts semantic similarity measurements in BERT's contextualized word embeddings. Using an identity probe and minimal bounding hypersphere analysis, it finds that high-frequency words have more diverse, less identifiable representations, leading to systematic over- or under-estimation of similarity—especially disadvantaging South American and African countries in embedding space, with distortions persisting even in multilingual BERT.

ABSTRACT

How does word frequency in pre-training data affect the behavior of similarity metrics in contextualized BERT embeddings? Are there systematic ways in which some word relationships are exaggerated or understated? In this work, we explore the geometric characteristics of contextualized word embeddings with two novel tools: (1) an identity probe that predicts the identity of a word using its embedding; (2) the minimal bounding sphere for a word's contextualized representations. Our results reveal that words of high and low frequency differ significantly with respect to their representational geometry. Such differences introduce distortions: when compared to human judgments, point estimates of embedding similarity (e.g., cosine similarity) can over- or under-estimate the semantic similarity of two words, depending on the frequency of those words in the training data. This has downstream societal implications: BERT-Base has more trouble differentiating between South American and African countries than North American and European ones. We find that these distortions persist when using BERT-Multilingual, suggesting that they cannot be easily fixed with additional data, which in turn introduces new distortions.

Motivation & Objective

• To investigate how word frequency in pre-training data affects the geometric structure of contextualized word embeddings.
• To examine whether these geometric differences lead to systematic distortions in semantic similarity estimation.
• To evaluate whether additional training data, such as in BERT-Multilingual, can mitigate these distortions.
• To highlight societal implications of frequency-based distortions in NLP models, especially regarding underrepresented regions.
• To advocate for transparency in model documentation, including word frequency and distortion profiles.

Proposed method

• Introduced an identity probe—a few-shot logistic regression classifier—to measure how identifiable a word is based on its contextualized embeddings.
• Defined the minimal bounding hypersphere around a word’s sibling embeddings to quantify representational diversity.
• Used BERT-base and BERT-Multilingual to extract contextualized embeddings from Wikipedia sentences, filtering for length and consistency in tokenization.
• Correlated word frequency (from Wikimedia and BookCorpus) with embedding geometry and similarity estimation errors.
• Evaluated distortion using the Stanford Contextualized Word Similarity dataset, comparing cosine similarity to human judgments.
• Conducted ablation studies to test the impact of context consistency and frequency on embedding richness.

Experimental results

Research questions

• RQ1 How does word frequency in pre-training data affect the geometric properties of contextualized word embeddings?
• RQ2 To what extent do frequency-based geometric differences distort semantic similarity estimates compared to human judgments?
• RQ3 Do multilingual pre-training data (as in BERT-Multilingual) reduce or eliminate these frequency-based distortions?
• RQ4 Are there measurable disparities in how country names from different continents are represented in embedding space?
• RQ5 Can the distortions observed be traced to differences in contextual information encoding based on word frequency?

Key findings

• High-frequency words in BERT have larger minimal bounding hyperspheres, indicating greater representational diversity, while low-frequency words are more identifiable and clustered.
• The identity probe shows that word identifiability decreases with increasing frequency, indicating that high-frequency words are harder to distinguish from one another.
• Cosine similarity between word embeddings correlates with human judgments only up to Pearson’s r = 0.42, with estimation errors systematically linked to word frequency.
• North American and European country names are represented with larger bounding spheres and are perceived as more distinct than South American and African country names, correlating with GDP and regional representation.
• BERT-Multilingual does not mitigate these distortions; the same frequency-based biases persist, suggesting that additional data alone cannot resolve the issue.
• The study concludes that all datasets contain distortions due to frequency and representational bias, and calls for transparency in model documentation to reveal such biases.