[Paper Review] A Comparison of Word Embeddings for English and Cross-Lingual Chinese Word Sense Disambiguation

Hong Jin Kang, Tao Chen+2 more|arXiv (Cornell University)|Nov 9, 2016

Topic Modeling25 references3 citations

TL;DR

This paper evaluates pretrained word embeddings—Word2Vec, GloVe, and Collobert & Weston—for monolingual English and cross-lingual English-Chinese Word Sense Disambiguation (WSD), enhancing the IMS WSD system with embedding features. It achieves state-of-the-art or competitive performance without retraining, introduces a public English-Chinese cross-lingual WSD dataset, and finds that simple embedding summation with proper scaling outperforms complex LSTM models.

ABSTRACT

Word embeddings are now ubiquitous forms of word representation in natural language processing. There have been applications of word embeddings for monolingual word sense disambiguation (WSD) in English, but few comparisons have been done. This paper attempts to bridge that gap by examining popular embeddings for the task of monolingual English WSD. Our simplified method leads to comparable state-of-the-art performance without expensive retraining. Cross-Lingual WSD - where the word senses of a word in a source language e come from a separate target translation language f - can also assist in language learning; for example, when providing translations of target vocabulary for learners. Thus we have also applied word embeddings to the novel task of cross-lingual WSD for Chinese and provide a public dataset for further benchmarking. We have also experimented with using word embeddings for LSTM networks and found surprisingly that a basic LSTM network does not work well. We discuss the ramifications of this outcome.

Motivation & Objective

• To compare the effectiveness of various pretrained word embeddings for monolingual English WSD across standard benchmarks (Senseval-2, Senseval-3, SemEval-2007).
• To extend the IMS WSD system by integrating word embeddings as features to improve performance without retraining.
• To investigate the feasibility and performance of word embeddings in cross-lingual WSD using English as source and Chinese as target language.
• To construct and release a gold-standard English-Chinese cross-lingual WSD dataset for educational applications and benchmarking.
• To evaluate the performance of a simple LSTM architecture with word embeddings for WSD and compare it with traditional supervised methods.

Proposed method

• Enhanced the open-source IMS WSD system by adding pretrained word embeddings (Word2Vec, GloVe, Collobert & Weston) as additional features.
• Applied a simple feature composition method: embedding vector summation with scaling parameters (σ = 0.05, 0.1, 0.15) to improve feature representation.
• Used the One-Million Sense-Tagged Instances dataset for training on All Words tasks in monolingual WSD.
• Constructed a new English-Chinese cross-lingual WSD dataset from real news articles, with human-annotated Chinese translations for target words.
• Evaluated performance using both fine-grained and coarse-grained scoring, with all annotations treated as correct for coarse-grained evaluation.
• Compared a basic LSTM network with word embeddings against the enhanced IMS system, using the same dataset and evaluation protocol.

Experimental results

Research questions

• RQ1 How do different pretrained word embeddings (Word2Vec, GloVe, Collobert & Weston) compare in performance for monolingual English WSD?
• RQ2 Can simple feature integration of word embeddings into an existing WSD system (IMS) achieve competitive or better results than state-of-the-art systems without retraining?
• RQ3 To what extent do word embeddings improve performance in cross-lingual WSD when translating English words into Chinese?
• RQ4 Why does a basic LSTM network fail to improve WSD performance when using word embeddings, despite their success in other NLP tasks?
• RQ5 How does the choice of scaling parameter (σ) affect the performance of embedded feature summation in WSD?

Key findings

• Adding pretrained word embeddings as features to the IMS system improved performance on monolingual WSD tasks, achieving competitive or better results than state-of-the-art systems without retraining.
• Word2Vec embeddings yielded the best performance, followed by GloVe and Collobert & Weston embeddings, across multiple benchmarks.
• The simple summation of embedding vectors with proper scaling (σ = 0.1) led to the highest accuracy (0.772) in cross-lingual WSD, outperforming the baseline by 21.3%.
• The performance gain from word embeddings was not statistically significant at 95% confidence due to the small size of the cross-lingual WSD dataset.
• A basic LSTM network with word embeddings performed poorly, suggesting that word embeddings alone are insufficient for effective sequence modeling in this WSD setup.
• Bing Translator’s phrase-level translation behavior led to poor alignment for single-word translations, undermining its reliability as a baseline for word-level cross-lingual WSD.