[Paper Review] Word Sense Disambiguation for 158 Languages using Word Embeddings Only

Varvara Logacheva, Денис Тесленко+8 more|MADOC (University of Mannheim)|Mar 14, 2020

Natural Language Processing Techniques37 references4 citations

TL;DR

This paper presents a fully unsupervised, knowledge-free method for word sense disambiguation (WSD) using only pre-trained fastText word embeddings. The proposed egvi (Ego-Graph Vector Induction) algorithm clusters word vectors based on semantic similarity to induce sense inventories, enabling WSD for 158 languages without external resources, with released models and an online system available for multilingual disambiguation.

ABSTRACT

Disambiguation of word senses in context is easy for humans, but is a major challenge for automatic approaches. Sophisticated supervised and knowledge-based models were developed to solve this task. However, (i) the inherent Zipfian distribution of supervised training instances for a given word and/or (ii) the quality of linguistic knowledge representations motivate the development of completely unsupervised and knowledge-free approaches to word sense disambiguation (WSD). They are particularly useful for under-resourced languages which do not have any resources for building either supervised and/or knowledge-based models. In this paper, we present a method that takes as input a standard pre-trained word embedding model and induces a fully-fledged word sense inventory, which can be used for disambiguation in context. We use this method to induce a collection of sense inventories for 158 languages on the basis of the original pre-trained fastText word embeddings by Grave et al. (2018), enabling WSD in these languages. Models and system are available online.

Motivation & Objective

• To develop a fully unsupervised WSD method that requires no external knowledge or annotated corpora.
• To enable word sense disambiguation for under-resourced languages lacking lexical resources or labeled training data.
• To create multilingual sense inventories from pre-trained fastText embeddings using graph clustering.
• To release a functional, multilingual WSD system capable of on-the-fly disambiguation across 158 languages.

Proposed method

• The method constructs a semantic similarity graph from pre-trained fastText word embeddings, where nodes are word vectors and edges represent cosine similarity.
• It applies ego-graph clustering to partition the graph into densely connected subgraphs, each representing a distinct word sense.
• Sense vectors are derived from cluster centroids, enabling disambiguation by assigning words to the nearest sense vector in context.
• The approach performs retrofitting of sense vectors using only the original word embeddings, avoiding retraining or external knowledge.
• The system supports on-the-fly WSD by identifying the language of input text and applying the corresponding pre-extracted sense inventory.
• The method is language-agnostic and leverages the distributional hypothesis to detect polysemy without supervision.

Experimental results

Research questions

• RQ1 Can word sense disambiguation be achieved without any external knowledge or labeled data using only pre-trained word embeddings?
• RQ2 How effective is graph clustering of word embeddings in inducing meaningful, coherent word sense inventories across diverse languages?
• RQ3 To what extent can unsupervised sense induction improve downstream NLP tasks in low-resource settings?
• RQ4 Can a single method scale to 158 languages using only pre-trained embeddings and no language-specific resources?
• RQ5 How does the performance of the induced sense vectors compare to non-disambiguated embeddings on multilingual similarity benchmarks?

Key findings

• The proposed egvi algorithm successfully induces sense inventories for 158 languages using only pre-trained fastText embeddings, with no external resources required.
• The system achieves improved performance on multilingual word similarity benchmarks compared to non-disambiguated word embeddings, demonstrating the utility of sense-level representations.
• The method is the first fully unsupervised WSD system for the majority of the 158 languages studied, particularly beneficial for under-resourced languages.
• Evaluation shows that sense vectors derived from clustering improve semantic similarity scores, validating the quality of the induced sense inventories.
• The released sense inventories and online system enable on-the-fly WSD across all 158 languages, supporting practical deployment in multilingual NLP pipelines.
• The approach outperforms baseline non-disambiguated embeddings on average, even without fine-tuning or supervision.