[Paper Review] Semi-supervised Word Sense Disambiguation with Neural Models

Dayu Yuan, Julian Richardson+3 more|arXiv (Cornell University)|Mar 22, 2016

Natural Language Processing TechniquesComputer Science37 references83 citations

TL;DR

This paper proposes a semi-supervised word sense disambiguation (WSD) framework that leverages Long Short-Term Memory (LSTM) networks to capture sequential and syntactic context, outperforming traditional Word2Vec-based methods. By combining an LSTM language model with label propagation on a graph of labeled and unlabeled sentences, the approach achieves state-of-the-art performance, especially on verbs, with a 10% improvement over strong baselines on SemEval all-words tasks.

ABSTRACT

Determining the intended sense of words in text - word sense disambiguation (WSD) - is a long standing problem in natural language processing. Recently, researchers have shown promising results using word vectors extracted from a neural network language model as features in WSD algorithms. However, a simple average or concatenation of word vectors for each word in a text loses the sequential and syntactic information of the text. In this paper, we study WSD with a sequence learning neural net, LSTM, to better capture the sequential and syntactic patterns of the text. To alleviate the lack of training data in all-words WSD, we employ the same LSTM in a semi-supervised label propagation classifier. We demonstrate state-of-the-art results, especially on verbs.

Motivation & Objective

• To address the challenge of limited labeled training data in all-words word sense disambiguation (WSD), particularly for large vocabularies.
• To improve WSD performance by capturing sequential and syntactic context using a recurrent neural network (LSTM) instead of static word embeddings.
• To leverage large-scale unlabeled text data through semi-supervised label propagation to enhance sense classification accuracy.
• To demonstrate that combining an LSTM language model with graph-based label propagation yields superior results compared to supervised baselines.

Proposed method

• Training a 1000-dimensional Word2Vec model on a 100-billion-word news corpus to generate initial word embeddings.
• Using an LSTM network to encode contextual sequences, capturing sequential and syntactic patterns more effectively than bag-of-words models.
• Constructing a graph where nodes represent sentences and edges are based on cosine similarity of LSTM-encoded sentence representations.
• Applying label propagation (LP) to transfer sense labels from a small set of labeled example sentences to a large set of unlabeled sentences via the graph.
• Using the LP-predicted labels as training data to refine the WSD classifier, improving decision boundaries and sense distribution estimation.
• Evaluating performance on SemEval benchmarks using F1 scores, comparing with Word2Vec baselines and most-frequent-sense models.

Experimental results

Research questions

• RQ1 Can an LSTM-based language model improve WSD performance compared to static word embeddings like Word2Vec, especially for verbs?
• RQ2 Does semi-supervised label propagation on a graph of labeled and unlabeled sentences enhance WSD accuracy when training data is scarce?
• RQ3 How does the quality of the language model (Word2Vec vs. LSTM) affect the performance of label propagation in WSD?
• RQ4 What impact does graph density and seed data composition have on the effectiveness of label propagation in WSD?
• RQ5 Can the proposed method generalize to other parts of speech and achieve consistent improvements across different word types?

Key findings

• The LSTM-based WSD model outperformed the Word2Vec baseline by 10% in F1 score on SemEval all-words tasks, with especially strong gains on verbs.
• Label propagation using the LSTM model improved F1 scores by 6.3 percentage points on SemCor and 7.3 percentage points on MASC compared to the Word2Vec baseline.
• The best performance was achieved when training data included both NOAD and SemCor or MASC, indicating that diverse seed data enhances label propagation.
• Graph density had a measurable effect: F1 scores remained stable between 85th and 98th percentiles of affinity thresholds, but dropped significantly at 80th percentile.
• The model achieved an F1 score of 0.872 on SemCor and 0.873 on MASC when using NOAD and SemCor/MASC as training data, demonstrating strong generalization.
• The approach showed robustness to varying numbers of unlabeled sentences per lemma, with no significant performance gain from increasing unlabeled data volume.