[Paper Review] On the Convergent Properties of Word Embedding Methods

Yingtao Tian, Vivek Kulkarni+2 more|arXiv (Cornell University)|May 12, 2016

Topic Modeling10 references3 citations

TL;DR

This paper proposes an intrinsic metric, ζ_CCA, to evaluate the consistency of word embedding methods across different random initializations by measuring canonical correlation between embedding dimensions. It finds that GloVe exhibits higher consistency and correlation with downstream task agreement than skip-gram, demonstrating that convergence stability varies significantly across methods and can be quantified without external data.

ABSTRACT

Do word embeddings converge to learn similar things over different initializations? How repeatable are experiments with word embeddings? Are all word embedding techniques equally reliable? In this paper we propose evaluating methods for learning word representations by their consistency across initializations. We propose a measure to quantify the similarity of the learned word representations under this setting (where they are subject to different random initializations). Our preliminary results illustrate that our metric not only measures a intrinsic property of word embedding methods but also correlates well with other evaluation metrics on downstream tasks. We believe our methods are is useful in characterizing robustness -- an important property to consider when developing new word embedding methods.

Motivation & Objective

• To investigate whether word embeddings converge to similar representations under different random initializations.
• To address the lack of intrinsic evaluation metrics for embedding consistency and robustness.
• To develop a method that quantifies similarity between embedding spaces independent of downstream tasks.
• To correlate intrinsic consistency with performance on standard NLP evaluation tasks like word analogy.

Proposed method

• Proposes a mapping-based similarity measure using one-to-one linear transformation between embedding dimensions to assess correspondence.
• Introduces a many-to-one alignment via Canonical Correlation Analysis (CCA) to model complex relationships between dimensions of two embedding matrices.
• Defines an intrinsic metric ζ_CCA as the average of the top canonical correlations between two embedding spaces.
• Applies CCA to compare GloVe and skip-gram embeddings trained with different random seeds and corpus orders.
• Uses Krippendorff’s alpha to compute agreement on the word analogy task as a benchmark for validation.
• Employs a combined corpus of 6 billion tokens (Wikipedia + News Crawl) with 400k vocabulary and 300-dimensional embeddings.

Experimental results

Research questions

• RQ1 Do word embeddings trained with different random initializations learn equivalent features, or is their performance consistency coincidental?
• RQ2 How can the consistency of learned word representations across initializations be quantitatively measured in an intrinsic way?
• RQ3 How do different embedding methods (e.g., GloVe vs. skip-gram) compare in terms of convergence stability and dimension-wise correspondence?
• RQ4 Does intrinsic consistency correlate with performance on downstream NLP tasks like the word analogy task?
• RQ5 Can canonical correlation analysis effectively capture the similarity between embedding spaces beyond simple correlation?

Key findings

• GloVe embeddings show higher consistency across random initializations than skip-gram, with a ζ_CCA of 0.74 compared to 0.66.
• The agreement score (Krippendorff’s alpha) on the word analogy task is 0.89 for GloVe-1/GloVe-2 and 0.79 for SG-1/SG-2, correlating with ζ_CCA.
• CCA-based canonical correlations reveal that GloVe achieves near-perfect correlation in the top 200 dimensions, indicating strong structural stability.
• The one-to-one dimension matching shows GloVe has 21.6% average correlation, compared to 19.5% for skip-gram, confirming greater consistency.
• The proposed ζ_CCA metric correlates strongly with model agreement on the analogy task, validating its usefulness as an intrinsic evaluation measure.
• Random embeddings yield near-zero correlations, confirming that observed similarities are not due to chance.