[Paper Review] Efficient Graph-based Word Sense Induction by Distributional Inclusion Vector Embeddings

Haw-Shiuan Chang, Amol Agrawal+5 more|arXiv (Cornell University)|Apr 9, 2018

Topic Modeling28 references3 citations

TL;DR

This paper proposes an efficient graph-based word sense induction method that uses distributional inclusion vector embeddings (DIVE) to create a global non-negative basis of topics, avoiding expensive nearest-neighbor searches. By clustering basis indexes in each word's ego network, the approach achieves state-of-the-art performance on WSI benchmarks while significantly improving computational efficiency over prior graph-based methods.

ABSTRACT

Word sense induction (WSI), which addresses polysemy by unsupervised discovery of multiple word senses, resolves ambiguities for downstream NLP tasks and also makes word representations more interpretable. This paper proposes an accurate and efficient graph-based method for WSI that builds a global non-negative vector embedding basis (which are interpretable like topics) and clusters the basis indexes in the ego network of each polysemous word. By adopting distributional inclusion vector embeddings as our basis formation model, we avoid the expensive step of nearest neighbor search that plagues other graph-based methods without sacrificing the quality of sense clusters. Experiments on three datasets show that our proposed method produces similar or better sense clusters and embeddings compared with previous state-of-the-art methods while being significantly more efficient.

Motivation & Objective

• To address the high computational cost of graph-based word sense induction (WSI) methods that rely on expensive nearest-neighbor searches.
• To improve the efficiency of WSI while maintaining or enhancing clustering quality for polysemous words.
• To enable the discovery of less common word senses by leveraging topic-based representations that adapt to target word context.
• To provide a scalable and interpretable framework for unsupervised word sense induction using topic-like basis vectors.
• To demonstrate that topic-based basis construction can replace word-level similarity computation without sacrificing performance.

Proposed method

• The method first learns a global non-negative vector embedding basis using distributional inclusion vector embeddings (DIVE), which compresses bag-of-words representations while preserving co-occurrence order.
• Each word is represented as a combination of basis vectors (topics), and for each polysemous word, the relevant basis indexes are identified based on DIVE's ability to model topic relevance to target words.
• A graph is constructed where nodes represent basis indexes (topics), and edges are weighted by target-dependent similarity between topics, computed using DIVE's context-aware topic-word probabilities.
• Graph clustering is applied to the ego network of each target word to discover sense clusters, with the number of clusters determined hierarchically.
• The method avoids full vocabulary nearest-neighbor search by operating on basis indexes instead of raw words, drastically reducing computational cost.
• The approach is evaluated using standard WSI benchmarks including TWSI and SemEval-2013, with sense prediction based on context embedding similarity.

Experimental results

Research questions

• RQ1 Can a topic-based basis representation replace word-level nearest-neighbor search in graph-based WSI while maintaining or improving performance?
• RQ2 Does using DIVE to compute target-dependent topic similarities enable more accurate and efficient sense clustering than traditional word-space methods?
• RQ3 Can the method effectively discover less common word senses without relying on predefined sense inventories?
• RQ4 How does the method scale in terms of computational efficiency compared to prior graph-based WSI approaches?
• RQ5 To what extent does the choice of DIVE dimensionality affect clustering quality and efficiency?

Key findings

• The proposed method achieves state-of-the-art performance on the TWSI benchmark, with an F1 score of 67.4 on balanced TWSI, outperforming the baseline WG method.
• On the SemEval-2013 task, the method achieves a Jaccard Index of 22.1 and Fuzzy NMI of 3.5, matching or exceeding other SOTA methods in key metrics.
• The method significantly reduces computational cost by replacing word-level nearest-neighbor search with basis index clustering, enabling efficient scaling to large vocabularies.
• DIVE-based basis learning allows context-aware topic selection, improving sense clustering accuracy by focusing on topics relevant to each target word.
• The method performs well even when the number of senses is not predetermined, supporting hierarchical clustering and multiple resolution levels.
• Experiments show that increasing DIVE dimensionality from 100 to 300 improves performance slightly, suggesting that higher-dimensional representations capture more nuanced sense distinctions.