[Paper Review] Unsupervised Lexical Substitution with Decontextualised Embeddings

Takashi Wada, Timothy Baldwin+2 more|arXiv (Cornell University)|Sep 17, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes an unsupervised lexical substitution method that retrieves synonyms using cosine similarity between contextualized embeddings of the target word and decontextualized embeddings averaged across multiple contexts. It outperforms strong baselines on English and Italian, especially for low-frequency words, and reduces morphosyntactic biases without fine-tuning or explicit supervision.

ABSTRACT

We propose a new unsupervised method for lexical substitution using pre-trained language models. Compared to previous approaches that use the generative capability of language models to predict substitutes, our method retrieves substitutes based on the similarity of contextualised and decontextualised word embeddings, i.e. the average contextual representation of a word in multiple contexts. We conduct experiments in English and Italian, and show that our method substantially outperforms strong baselines and establishes a new state-of-the-art without any explicit supervision or fine-tuning. We further show that our method performs particularly well at predicting low-frequency substitutes, and also generates a diverse list of substitute candidates, reducing morphophonetic or morphosyntactic biases induced by article-noun agreement.

Motivation & Objective

• To address limitations in generative language model approaches to lexical substitution, such as poor performance on low-frequency words and morphosyntactic biases from context.
• To develop a fully unsupervised method that avoids fine-tuning and explicit supervision while leveraging pre-trained models.
• To improve synonym retrieval by using averaged, decontextualized word embeddings derived from multiple contexts.
• To reduce biases from article–noun agreement and morphophonetic constraints in English and Italian.
• To demonstrate state-of-the-art performance on lexical substitution in both English and Italian with minimal architectural changes.

Proposed method

• The method computes contextualized representations of the target word in context using a pre-trained transformer model (e.g., BERT), summing representations across selected layers to form f(x,c).
• For each candidate substitute y, it computes a decontextualized embedding f(y) as the average of its contextualized representations across N sentences from a monolingual corpus.
• Substitute candidates are ranked by cosine similarity between f(x,c) and f(y), using the formula S(y|x,c) = cos(f(y), f(x,c)).
• To handle polysemy, the method clusters N sentences containing y into K groups based on embedding similarity, then computes a separate decontextualized embedding f^k(y) per cluster.
• The method pre-computes f(y) for all words in a custom vocabulary Ṽ that includes subword-segmented low-frequency words, enabling OOV word substitution.
• The approach uses layer aggregation (summing over layers) rather than weighted combination, as it showed no significant improvement.

Experimental results

Research questions

• RQ1 Can a fully unsupervised method based on embedding similarity outperform generative approaches in lexical substitution without fine-tuning?
• RQ2 Does using decontextualized embeddings improve performance on low-frequency or OOV words compared to masked language model generation?
• RQ3 To what extent does the method reduce morphosyntactic biases, such as those induced by article–noun agreement in English and Italian?
• RQ4 How does the performance vary across different transformer layers, and which layers yield the best representations for lexical substitution?
• RQ5 Does multi-sense clustering of word representations improve synonym retrieval for polysemous words?

Key findings

• The proposed method achieves a new state-of-the-art on both English and Italian lexical substitution benchmarks without any fine-tuning or explicit supervision.
• On the SWORDS dataset, the method with a 30k vocabulary achieved 39.9 F1, outperforming BERT-K (30.4) and DeBERTa (39.9), demonstrating strong generalization.
• The method predicts nearly twice as many low-frequency substitutes (frequency < 50k) as the BERT-K baseline when using a 30k vocabulary.
• With a 10k vocabulary, the method still outperforms BERT-K in Fc (32.6 vs. 28.1), confirming effectiveness even with limited vocabulary size.
• The method reduces morphosyntactic bias: it avoids over-reliance on preceding articles (e.g., 'an increase' leading to vowel-initial words) and generates more semantically diverse substitutes.
• Using multi-sense clustering improves performance slightly, with Fc increasing from 36.0 (K=1) to 36.9 (K=8), though gains plateau beyond K=8.