[Paper Review] AutoExtend: Extending Word Embeddings to Embeddings for Synsets and Lexemes

Sascha Rothe, Hinrich Schütze|arXiv (Cornell University)|Jul 4, 2015

Topic ModelingComputer Science43 references204 citations

TL;DR

AutoExtend is a flexible method that extends pre-trained word embeddings to embeddings for synsets and lexemes using a tensor-based formalization of lexical resource constraints. By modeling words as sums of their lexemes and synsets as sums of their lexemes, it learns shared vector space representations without retraining, achieving state-of-the-art performance on word similarity and word sense disambiguation tasks.

ABSTRACT

We present extit{AutoExtend}, a system to learn embeddings for synsets and lexemes. It is flexible in that it can take any word embeddings as input and does not need an additional training corpus. The synset/lexeme embeddings obtained live in the same vector space as the word embeddings. A sparse tensor formalization guarantees efficiency and parallelizability. We use WordNet as a lexical resource, but AutoExtend can be easily applied to other resources like Freebase. AutoExtend achieves state-of-the-art performance on word similarity and word sense disambiguation tasks.

Motivation & Objective

• To extend pre-trained word embeddings to embeddings for non-word entities like synsets and lexemes in lexical resources such as WordNet.
• To develop a method that operates in the same vector space as input word embeddings, enabling direct compatibility with downstream NLP tasks.
• To avoid the need for additional training corpora or retraining by leveraging structural constraints from lexical resources.
• To achieve state-of-the-art performance on word similarity and word sense disambiguation benchmarks.
• To provide a generalizable framework applicable to other knowledge bases like Freebase and multilingual resources.

Proposed method

• Formalizes word, lexeme, and synset embeddings using tensor equations, where word vectors are sums of their lexeme vectors and synset vectors are sums of their lexeme vectors.
• Introduces a learnable diagonal transformation matrix E(i,j) per word-synset pair to distribute word embeddings to lexemes, with each dimension treated independently for efficiency and parallelism.
• Imposes the constraint that the sum of transformation matrices per word equals the identity matrix, ensuring vector conservation across lexemes.
• Solves the system using a regularized least-squares optimization over the tensor formalization, minimizing reconstruction error for words and synsets.
• Applies a weighted combination of three constraints: word reconstruction, synset reconstruction, and lexeme consistency, with hyperparameter tuning for optimal performance.
• Supports multilingual extension by learning a linear transformation matrix L to align embeddings across languages, enabling cross-lingual synset embedding computation.

Experimental results

Research questions

• RQ1 Can we extend pre-trained word embeddings to embeddings for synsets and lexemes without retraining or additional corpora?
• RQ2 How well can a method based on structural constraints from WordNet perform on standard NLP benchmarks like word similarity and word sense disambiguation?
• RQ3 What is the optimal balance between different constraints (word, synset, lexeme) in the optimization objective?
• RQ4 Can the method be generalized to other knowledge bases like Freebase or multilingual resources?
• RQ5 How does AutoExtend compare to existing methods that train sense-specific embeddings from scratch?

Key findings

• AutoExtend achieves state-of-the-art performance on the SCWS word similarity benchmark, outperforming prior methods that use sense-specific embeddings.
• On the word sense disambiguation task, AutoExtend significantly outperforms the naive baseline (Snaive) that simply averages word vectors for a synset.
• The method achieves competitive performance on WSD when using only the word embeddings and resource constraints, without any additional training data.
• The optimal weighting of constraints (word, synset, lexeme) is found to be near equal importance, with minimal performance drop for a wide range of values (θ ∈ [0.2, 0.8]) in the weighted combination.
• The approach generalizes well to multilingual settings by aligning embeddings across languages using a linear transformation matrix, enabling cross-lingual synset embeddings.
• The authors release the AutoExtend code, pre-trained lexeme and synset embeddings, and evaluation scripts, enabling full reproducibility of results.