[Paper Review] Angular-Based Word Meta-Embedding Learning.

James O’Neill, Danushka Bollegala|arXiv (Cornell University)|Aug 13, 2018

Topic Modeling17 references3 citations

TL;DR

This paper proposes an angular-based meta-embedding approach that improves word representation learning by optimizing for cosine similarity and Kullback-Leibler divergence instead of standard ℓ₁ and ℓ₂ losses. It demonstrates that normalization-aware objectives outperform traditional methods on word similarity and relatedness benchmarks, achieving superior performance over existing meta-learning strategies in unsupervised settings.

ABSTRACT

Ensembling word embeddings to improve distributed word representations has shown good success for natural language processing tasks in recent years. These approaches either carry out straightforward mathematical operations over a set of vectors or use unsupervised learning to find a lower-dimensional representation. This work compares meta-embeddings trained for different losses, namely loss functions that account for angular distance between the reconstructed embedding and the target and those that account normalized distances based on the vector length. We argue that meta-embeddings are better to treat the ensemble set equally in unsupervised learning as the respective quality of each embedding is unknown for upstream tasks prior to meta-embedding. We show that normalization methods that account for this such as cosine and KL-divergence objectives outperform meta-embedding trained on standard $\ell_1$ and $\ell_2$ loss on extit{defacto} word similarity and relatedness datasets and find it outperforms existing meta-learning strategies.

Motivation & Objective

• To improve distributed word representations by ensembling multiple pre-trained word embeddings through meta-embedding learning.
• To address the challenge of unknown relative quality of individual embeddings in unsupervised settings prior to downstream tasks.
• To investigate whether normalization-aware loss functions—such as cosine similarity and KL-divergence—yield better meta-embeddings than standard ℓ₁ and ℓ₂ losses.
• To evaluate the effectiveness of angular distance-based objectives in capturing semantic relationships in word representations.
• To compare the proposed method against existing meta-learning strategies in unsupervised word embedding ensembling.

Proposed method

• The method employs meta-embedding learning to combine multiple pre-trained word embeddings into a single, unified representation.
• It introduces a loss function based on angular distance, specifically using cosine similarity to measure the alignment between the reconstructed embedding and target vectors.
• The approach incorporates Kullback-Leibler (KL) divergence as an alternative normalization-aware objective to account for vector length and directional similarity.
• The model is trained using these angular-based objectives instead of standard ℓ₁ or ℓ₂ loss functions.
• The method treats all input embeddings equally during training, assuming no prior knowledge of their individual quality in upstream tasks.
• The final meta-embedding is optimized to preserve semantic similarity and relatedness by minimizing angular and normalized distance metrics.

Experimental results

Research questions

• RQ1 Does using angular distance-based loss functions such as cosine similarity improve meta-embedding quality compared to standard ℓ₁ and ℓ₂ losses?
• RQ2 How do normalization-aware objectives like KL-divergence compare to non-normalized losses in capturing semantic relationships in word embeddings?
• RQ3 Can angular-based meta-embedding learning outperform existing meta-learning strategies in unsupervised word representation learning?
• RQ4 To what extent do normalization-aware objectives mitigate the impact of varying vector magnitudes in ensemble learning?
• RQ5 How does the proposed method perform on standard word similarity and relatedness evaluation benchmarks?

Key findings

• Normalization-aware loss functions such as cosine similarity and KL-divergence outperform standard ℓ₁ and ℓ₂ losses in meta-embedding training.
• The proposed angular-based meta-embedding approach achieves better performance on defacto word similarity and relatedness datasets compared to baseline methods.
• The method demonstrates consistent improvement over existing meta-learning strategies in unsupervised word embedding ensembling.
• The use of cosine and KL-divergence objectives leads to more robust and semantically aligned meta-embeddings.
• The results confirm that treating the ensemble set equally without prior knowledge of embedding quality is effective when using angular and normalized distance metrics.
• The method shows strong generalization on benchmark datasets, indicating its effectiveness for downstream NLP tasks.