[Paper Review] What do you mean, BERT? Assessing BERT as a Distributional Semantics Model

Timothee Mickus, Denis Paperno+2 more|arXiv (Cornell University)|Nov 13, 2019

Topic Modeling57 references60 citations

TL;DR

The paper evaluates whether BERT’s contextual embeddings form a coherent distributional semantic space, examining word-type cohesion and cross-sentence effects introduced by BERT’s segmentation and next-sentence prediction objective.

ABSTRACT

Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous noncontextual distributional semantic models. In this work, we focus on BERT, a deep neural network that produces contextualized embeddings and has set the state-of-the-art in several semantic tasks, and study the semantic coherence of its embedding space. While showing a tendency towards coherence, BERT does not fully live up to the natural expectations for a semantic vector space. In particular, we find that the position of the sentence in which a word occurs, while having no meaning correlates, leaves a noticeable trace on the word embeddings and disturbs similarity relationships.

Motivation & Objective

• Assess how well BERT’s contextual embeddings align with distributional semantics expectations.
• Test whether identical word types form coherent clusters in BERT’s embedding space.
• Investigate cross-sentence coherence and segment-induced biases in BERT representations.

Proposed method

• Treat BERT as a black box and extract last-layer embeddings for tokens.
• Use silhouette scores to assess word-type cohesion across token embeddings.
• Analyze segment encodings and residual connections to measure cross-sentence biases.
• Compute mean-squared error (MSE) to test coherence of token groups across segments.
• Evaluate sentence-level coherence by comparing cosine similarities within sentences and across consecutive sentences.

Experimental results

Research questions

• RQ1 Do word types form natural, cohesive clusters in BERT’s contextual embedding space?
• RQ2 Does BERT introduce semantically irrelevant differences between tokens from different sentence segments (segment A vs segment B) and how does this affect coherence?
• RQ3 How does segment encoding and positional encoding influence cross-sentence and intra-sentence semantic relationships in BERT?
• RQ4 Do sentence-level representations produced by BERT exhibit coherent semantics suitable for distributional semantics benchmarks?

Key findings

• Word-type cohesion in BERT is present but weak: about 25.9% of tokens have negative silhouette scores, and 10% of types contain only negative scores.
• Cosine similarity between average BERT embeddings of word pairs correlates with human similarity ratings (Spearman 0.705), outperforming a Word2Vec baseline (0.669).
• Cross-sentence coherence exhibits a significant segment effect, with tokens from different segments showing systematic bias due to segment encodings and residuals; effect size is moderate (d = -0.527).
• For many items, embeddings from the same type across segments are more coherent with their segment mean than with the other segment’s mean, indicating partial preservation of segment biases.
• When using a one-sentence input scheme, BERT’s sentence representations achieve competitive correlations on STS and SICK-R benchmarks, but under a two-sentence input scheme performance degrades below Word2Vec.