[Paper Review] Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift

Matej Martinc, Petra Kralj Novak+1 more|arXiv (Cornell University)|Dec 2, 2019

Language and cultural evolutionSocial Sciences28 references42 citations

TL;DR

The paper introduces a method that uses time-specific Word representations from fine-tuned BERT to detect diachronic semantic shift, showing comparable performance to state-of-the-art on LiverpoolFC, and applying to Brexit and multilingual immigration corpora.

ABSTRACT

We propose a new method that leverages contextual embeddings for the task of diachronic semantic shift detection by generating time specific word representations from BERT embeddings. The results of our experiments in the domain specific LiverpoolFC corpus suggest that the proposed method has performance comparable to the current state-of-the-art without requiring any time consuming domain adaptation on large corpora. The results on the newly created Brexit news corpus suggest that the method can be successfully used for the detection of a short-term yearly semantic shift. And lastly, the model also shows promising results in a multilingual settings, where the task was to detect differences and similarities between diachronic semantic shifts in different languages.

Motivation & Objective

• Motivate and measure semantic drift over short time spans using contextual embeddings.
• Create time-specific word representations by fine-tuning BERT on time-segmented corpora.
• Evaluate the approach on domain-specific and multilingual corpora to assess generality and efficiency.

Proposed method

• Fine-tune a pretrained BERT model on each time-segment corpus for domain adaptation without diachronic fine-tuning.
• Generate time-specific sequence embeddings by feeding byte-pair encoded sequences into BERT and summing the last four encoder layers.
• Create word-level time representations by aggregating contextual embeddings for all token occurrences in a time period.
• Reconstruct word representations when Byte-Pair Tokens split a word by averaging constituent token embeddings.
• Measure semantic shift via cosine distance between time-specific representations of the same token.
• For Brexit, analyze relative shifts by tracking cosine similarity between Brexit and 50 semantically related words filtered by normalized Levenshtein distance.

Experimental results

Research questions

• RQ1 Can contextual embeddings from time-tuned BERT detect short-term semantic shifts without large-domain adaptation?
• RQ2 Do time-specific representations enable cross-language comparisons of semantic shift in multilingual corpora?
• RQ3 How do time-specific representations correlate with human-annotated semantic shift on benchmarks like LiverpoolFC?
• RQ4 Are detected shifts in Brexit and immigration discourse aligned with real-world events and discourse changes?

Key findings

• Performance on LiverpoolFC shows Pearson correlation of 0.47 between cosine distance and semantic shift index, close to the state-of-the-art.
• The method achieves comparable results to prior work without large-scale domain adaptation.
• Brexit corpus analysis reveals meaningful changes in word relations over years that align with events and debates.
• Multilingual immigration analysis (English-Slovenian) shows that English and Slovenian word pairs largely share semantic trajectories with immigration discourse.
• The approach enables analysis in languages with limited resources using a single multilingual BERT model.
• The method demonstrates potential for short-term semantic shift detection and cross-language comparisons.