[Paper Review] Incorporating Word Sense Disambiguation in Neural Language Models

Jan Philip Wahle, Terry Ruas+2 more|arXiv (Cornell University)|Jun 15, 2021

Natural Language Processing Techniques27 references4 citations

TL;DR

This paper proposes two supervised pre-training methods—LMGC and LMGC-M—that integrate WordNet gloss definitions into neural language models to enhance word sense disambiguation (WSD) without adding significant parameters. By jointly training on masked language modeling and gloss classification, the approach improves WSD performance by 0.5% F1 on SemEval and Senseval benchmarks and boosts general language understanding by 1.1% on average across GLUE tasks, outperforming state-of-the-art models including GlossBERT and KBERT-W+W.

ABSTRACT

We present two supervised (pre-)training methods to incorporate gloss definitions from lexical resources into neural language models (LMs). The training improves our models' performance for Word Sense Disambiguation (WSD) but also benefits general language understanding tasks while adding almost no parameters. We evaluate our techniques with seven different neural LMs and find that XLNet is more suitable for WSD than BERT. Our best-performing methods exceeds state-of-the-art WSD techniques on the SemCor 3.0 dataset by 0.5% F1 and increase BERT's performance on the GLUE benchmark by 1.1% on average.

Motivation & Objective

• To improve word sense disambiguation (WSD) in neural language models by integrating lexical knowledge from WordNet glosses during pre-training.
• To maintain or enhance the model's performance on downstream NLP tasks beyond WSD, such as text similarity and sentence classification.
• To develop a general, parameter-efficient method applicable to any transformer-based language model, avoiding costly retraining or architectural modifications.
• To demonstrate that WSD pre-training enhances general language understanding, as measured by improved performance on benchmarks like GLUE.
• To provide a publicly available implementation to support reproducibility and further research in knowledge-augmented language modeling.

Proposed method

• The method, Language Model Gloss Classification (LMGC), treats WSD as a text classification task by concatenating a sentence with a candidate gloss from WordNet, separated by a special token.
• The model is fine-tuned with two supervision signals: highlighting the ambiguous word with special tokens and prepending the polysemous word before the gloss definition.
• A linear classifier head is applied to the [CLS] token representation to predict the correct gloss, leveraging the model’s contextual embeddings.
• The enhanced method, LMGC-M, combines masked language modeling (MLM) and gloss classification in a single training pass, improving representation learning through joint optimization.
• The approach is applied to various pre-trained models (e.g., BERT, RoBERTa, XLNet), with no modifications to word embeddings or attention mechanisms.
• The method avoids parameter-heavy mechanisms like knowledge attention or word-piece re-tokenization, ensuring efficiency and scalability.

Experimental results

Research questions

• RQ1 Can integrating WordNet gloss definitions during pre-training improve word sense disambiguation performance in neural language models?
• RQ2 Does pre-training on WSD with glosses enhance the model’s generalization to downstream NLP tasks beyond WSD?
• RQ3 How does the joint training of MLM and gloss classification (LMGC-M) compare to separate or sequential fine-tuning in terms of performance and efficiency?
• RQ4 Can a lightweight, parameter-efficient method outperform heavier models like KBERT-W+W that use complex attention mechanisms and 4x more parameters?
• RQ5 To what extent does WSD pre-training improve performance on benchmarks like GLUE and SuperGLUE?

Key findings

• LMGC-M achieves a 0.5% absolute improvement in F1 score over the best-performing baseline (GlossBERT) on SemEval and Senseval WSD benchmarks.
• The method improves average performance on the GLUE benchmark by 1.1% compared to the BERT BASE baseline, demonstrating enhanced general language understanding.
• LMGC-M outperforms KBERT-W+W, which uses approximately four times more parameters and 32% more training time, while maintaining similar inference efficiency.
• The joint training of MLM and gloss classification (LMGC-M) leads to better performance than separate training, suggesting synergistic learning of semantic and syntactic representations.
• XLNet-based LMGC-M achieves the highest WSD performance among all models tested, outperforming BERT and RoBERTa, even though it has no additional parameters.
• The method generalizes well: models pre-trained with WSD show consistent improvements across diverse NLP tasks, including text similarity, sentence classification, and natural language inference.