[Paper Review] Sense Vocabulary Compression through the Semantic Knowledge of WordNet for Neural Word Sense Disambiguation

Loïc Vial, Benjamin Lecouteux+1 more|arXiv (Cornell University)|May 14, 2019

Natural Language Processing Techniques24 references8 citations

TL;DR

This paper proposes two sense vocabulary compression methods that leverage WordNet's semantic relationships—such as hypernymy and hyponymy—to reduce the number of sense tags in word sense disambiguation (WSD) systems. By focusing only on essential senses, the approach reduces model size by 1.2–2×, improves coverage to 99.99% on SemCor, and enables state-of-the-art performance using BERT embeddings without additional training data.

ABSTRACT

In this article, we tackle the issue of the limited quantity of manually sense annotated corpora for the task of word sense disambiguation, by exploiting the semantic relationships between senses such as synonymy, hypernymy and hyponymy, in order to compress the sense vocabulary of Princeton WordNet, and thus reduce the number of different sense tags that must be observed to disambiguate all words of the lexical database. We propose two different methods that greatly reduces the size of neural WSD models, with the benefit of improving their coverage without additional training data, and without impacting their precision. In addition to our method, we present a WSD system which relies on pre-trained BERT word vectors in order to achieve results that significantly outperform the state of the art on all WSD evaluation tasks.

Motivation & Objective

• Address the scarcity of manually annotated WSD corpora by reducing the number of required sense tags.
• Improve generalization and coverage of supervised WSD models without additional training data.
• Eliminate the need for backoff strategies in WSD by ensuring near-complete sense coverage using compressed sense inventories.
• Achieve state-of-the-art performance in WSD using pre-trained contextual embeddings like BERT.
• Develop a reusable, documented tool and model for training and evaluating neural WSD systems.

Proposed method

• Construct a compressed sense vocabulary by selecting only essential senses—specifically, hypernyms and broader semantic relations—using WordNet's semantic graph.
• Apply two compression strategies: one based on hypernyms only, and another using all semantic relations (hypernyms, hyponyms, meronyms, etc.).
• Train a neural WSD model using BERT, ELMo, or GloVe embeddings on a combination of SemCor and WordNet glosses (WNGC) to improve contextual representation.
• Use ensembling of eight independently trained models to improve robustness and performance, with final predictions obtained by averaging class probabilities.
• Leverage the semantic hierarchy in WordNet to group semantically related senses, allowing a single sense to represent a cluster of related meanings.
• Apply the compressed sense vocabulary to both training and inference, enabling the model to disambiguate words even if they were not seen in the original training data.

Experimental results

Research questions

• RQ1 Can semantic relationships in WordNet be used to compress the sense vocabulary without sacrificing disambiguation accuracy?
• RQ2 To what extent can sense vocabulary compression improve the coverage of supervised WSD models on unseen words?
• RQ3 Does using a compressed sense vocabulary reduce model size while maintaining or improving performance?
• RQ4 How does the integration of BERT embeddings with sense vocabulary compression affect WSD performance across standard benchmarks?
• RQ5 Can the proposed method eliminate the need for backoff strategies in WSD systems?

Key findings

• The proposed sense vocabulary compression methods reduce the total number of sense tags in WordNet to just 6% of the original size.
• Coverage of the SemCor training corpus increases from ~16% to over 99.99% when using the compressed sense vocabulary, with only one word missing from the evaluation set.
• When combined with WNGC (WordNet glosses), the system achieves 100% coverage on polysemic words in evaluation tasks.
• The use of BERT embeddings improves the F1 score on the 'ALL' task to 79.00% when using hypernym-based compression and ensembling, significantly outperforming prior state-of-the-art methods.
• The model size is reduced by a factor of 1.2 to 2× due to the compressed sense vocabulary, without any loss in precision.
• Ablation studies confirm that BERT embeddings and the inclusion of WNGC data have the largest positive impact on performance, with ensembling adding approximately 1 point to the F1 score.