[Paper Review] sense2vec - A Fast and Accurate Method for Word Sense Disambiguation In Neural Word Embeddings

Andrew Trask, Phil Michalak+1 more|arXiv (Cornell University)|Nov 19, 2015

Natural Language Processing Techniques13 references137 citations

TL;DR

This paper introduces sense2vec, a fast and accurate method for word sense disambiguation in neural word embeddings by leveraging supervised Part-of-Speech tagging to assign context-specific embeddings. It achieves over 8% average error reduction in unlabeled attachment scores across six languages in neural dependency parsing, demonstrating that sense-disambiguated embeddings significantly improve syntactic parsing performance compared to standard single-vector models.

ABSTRACT

Neural word representations have proven useful in Natural Language Processing (NLP) tasks due to their ability to efficiently model complex semantic and syntactic word relationships. However, most techniques model only one representation per word, despite the fact that a single word can have multiple meanings or "senses". Some techniques model words by using multiple vectors that are clustered based on context. However, recent neural approaches rarely focus on the application to a consuming NLP algorithm. Furthermore, the training process of recent word-sense models is expensive relative to single-sense embedding processes. This paper presents a novel approach which addresses these concerns by modeling multiple embeddings for each word based on supervised disambiguation, which provides a fast and accurate way for a consuming NLP model to select a sense-disambiguated embedding. We demonstrate that these embeddings can disambiguate both contrastive senses such as nominal and verbal senses as well as nuanced senses such as sarcasm. We further evaluate Part-of-Speech disambiguated embeddings on neural dependency parsing, yielding a greater than 8% average error reduction in unlabeled attachment scores across 6 languages.

Motivation & Objective

• To address the limitation of single-vector word embeddings that conflate multiple word senses into a single superposition, which harms downstream NLP performance.
• To reduce the computational cost of word sense modeling by replacing unsupervised clustering with supervised labeling, enabling faster training and inference.
• To improve the accuracy of neural syntactic parsing by providing context-specific, sense-disambiguated word embeddings.
• To evaluate whether sense-disambiguated embeddings outperform standard word embeddings in multilingual dependency parsing tasks.
• To demonstrate that supervised disambiguation enables efficient and effective selection of appropriate word vectors for NLP models.

Proposed method

• The method uses a pre-trained word embedding model and applies a supervised Part-of-Speech tagger to label each word occurrence with its grammatical sense.
• For each word, context embeddings are computed as a weighted average of surrounding words using tf-idf weighting.
• These context embeddings are clustered per word to identify distinct sense prototypes, with cluster labels assigned via supervised POS tagging.
• Each word occurrence is re-labeled with its corresponding sense cluster, and a new word embedding model is trained using the structured skip-gram approach with sense-specific labels.
• The final embeddings are trained using identical hyperparameters to baseline models, ensuring fair comparison.
• The approach integrates sense-specific embeddings directly into a neural dependency parser that uses gold-standard POS tags as input indices.

Experimental results

Research questions

• RQ1 Can supervised disambiguation of word senses using POS tags produce more accurate and efficient word embeddings than traditional single-vector models?
• RQ2 Does the use of sense-disambiguated embeddings lead to measurable improvements in syntactic parsing performance across multiple languages?
• RQ3 How does the computational cost of sense2vec compare to unsupervised clustering-based word sense models?
• RQ4 To what extent do sense-disambiguated embeddings improve the disambiguation of contrastive and nuanced word senses, such as nominal vs. verbal or sarcasm?
• RQ5 Can the proposed method be generalized to other types of supervised labels beyond POS tags?

Key findings

• sense2vec achieved an average error reduction of 8.52% in unlabeled attachment scores across six languages in dependency parsing, with reductions ranging from 3.98% to 13.69% per language.
• The method reduced parsing error by 12.71% in Swedish and 13.69% in German, indicating strong performance gains in morphologically rich languages.
• The model outperformed baseline wang2vec embeddings on all six languages, with absolute error reductions ranging from 2.47% to 14.54%.
• The use of sense2vec embeddings led to a 5.17% error reduction in Bulgarian and 10.93% in German, demonstrating consistent gains across diverse linguistic structures.
• The approach maintained high performance even when removing malformed tokens from treebanks, indicating robustness in real-world NLP pipelines.
• The results confirm that disambiguating word senses into separate embeddings alleviates the superposition problem and improves downstream NLP model accuracy.