[Paper Review] Inducing and Embedding Senses with Scaled Gumbel Softmax

Fenfei Guo, Mohit Iyyer+1 more|arXiv (Cornell University)|Apr 22, 2018

Neural Networks and Applications79 references3 citations

TL;DR

This paper proposes Gumbel Attention for Sense Induction (gasi), a differentiable, attention-based model that uses scaled Gumbel-Softmax to induce discrete, interpretable word senses. Unlike prior methods, gasi achieves competitive word similarity scores while producing more coherent and human-interpretable sense representations, as validated by a new human-centric coherence evaluation.

ABSTRACT

Methods for learning word sense embeddings represent a single word with multiple sense-specific vectors. These methods should not only produce interpretable sense embeddings, but should also learn how to select which sense to use in a given context. We propose an unsupervised model that learns sense embeddings using a modified Gumbel softmax function, which allows for differentiable discrete sense selection. Our model produces sense embeddings that are competitive (and sometimes state of the art) on multiple similarity based downstream evaluations. However, performance on these downstream evaluations tasks does not correlate with interpretability of sense embeddings, as we discover through an interpretability comparison with competing multi-sense embeddings. While many previous approaches perform well on downstream evaluations, they do not produce interpretable embeddings and learn duplicated sense groups; our method achieves the best of both worlds.

Motivation & Objective

• To address the gap between computer-centric evaluations and human interpretability in multisense word embeddings.
• To develop a new evaluation metric that measures how well sense representations support human understanding of word meanings.
• To design a model that prioritizes interpretability and coherence for human users, not just downstream NLP performance.
• To demonstrate that simpler, discrete sense induction models can outperform complex contextual models in human-in-the-loop linguistic tasks.
• To bridge the disconnect between representation learning and lexical semantics by aligning sense induction with human-usable sense inventories.

Proposed method

• Uses a Word2Vec-inspired objective to jointly learn context embeddings C and sense embeddings S for each word.
• Applies Gumbel-Softmax with temperature scaling to differentiably sample discrete word senses from a set of K candidate senses.
• Employs an attention mechanism to select the most relevant sense based on local context, enabling hard, interpretable sense assignments.
• Introduces a pruning strategy using cosine distance thresholds to remove duplicate or near-identical senses.
• Optimizes the model using a likelihood objective over context words, marginalized over all possible senses.
• Uses a soft-attention baseline (sasi) as a stepping-stone to validate the effectiveness of the discrete Gumbel-Softmax approach.

Experimental results

Research questions

• RQ1 Can a differentiable, discrete sense induction model produce sense representations that are more interpretable to humans than existing models?
• RQ2 How do computer-centric evaluations (e.g., word similarity) correlate with human-centric interpretability of word senses?
• RQ3 Does a simpler, attention-based model with discrete sense selection outperform more complex, end-to-end contextual models in human-in-the-loop tasks?
• RQ4 To what extent can Gumbel-Softmax with temperature scaling improve the coherence and distinctiveness of induced word senses?
• RQ5 How can sense induction models be better aligned with linguistic tasks such as sense inventory construction and semantic drift analysis?

Key findings

• The proposed coherence evaluation successfully exposes that existing multisense models often fail to produce human-interpretable sense representations despite strong computer-centric performance.
• gasi achieves competitive performance on both contextual and non-contextual word similarity benchmarks, outperforming prior models in interpretability.
• gasi produces more distinguishable and coherent sense representations than existing methods, as measured by the new human-centric coherence metric.
• The model’s use of differentiable Gumbel-Softmax enables hard, discrete sense selection while maintaining full training differentiability, improving interpretability.
• Pruning based on cosine distance between sense embeddings effectively removes redundant or duplicated senses, improving sense clarity.
• The results suggest that future sense induction should be guided by human-driven, task-specific metrics rather than purely automatic evaluation metrics.