[Paper Review] A Resource-Free Evaluation Metric for Cross-Lingual Word Embeddings Based on Graph Modularity

Yoshinari Fujinuma, Jordan Boyd‐Graber+1 more|arXiv (Cornell University)|Jun 5, 2019

Topic Modeling54 references21 citations

TL;DR

This paper proposes a resource-free intrinsic evaluation metric for cross-lingual word embeddings based on graph modularity, which measures how well words from different languages are mixed in the embedding space. The method constructs a k-nearest neighbor graph from word vectors and computes modularity to detect language-specific clustering; it shows strong negative correlation with downstream task performance and improves unsupervised cross-lingual embedding selection, especially on low-resource, distant language pairs.

ABSTRACT

Cross-lingual word embeddings encode the meaning of words from different languages into a shared low-dimensional space. An important requirement for many downstream tasks is that word similarity should be independent of language - i.e., word vectors within one language should not be more similar to each other than to words in another language. We measure this characteristic using modularity, a network measurement that measures the strength of clusters in a graph. Modularity has a moderate to strong correlation with three downstream tasks, even though modularity is based only on the structure of embeddings and does not require any external resources. We show through experiments that modularity can serve as an intrinsic validation metric to improve unsupervised cross-lingual word embeddings, particularly on distant language pairs in low-resource settings.

Motivation & Objective

• To develop an intrinsic evaluation metric for cross-lingual word embeddings that does not require external linguistic resources.
• To address the lack of reliable intrinsic metrics in low-resource and low-resource language settings.
• To improve the robustness of unsupervised cross-lingual embedding models by using modularity as a validation criterion.
• To demonstrate that modularity captures complementary information compared to existing intrinsic metrics like QVEC-CCA and CSLS.

Proposed method

• Constructs a k-nearest neighbor graph from cross-lingual word embeddings using vector similarity.
• Applies the modularity measure from network science to quantify the strength of language-specific clusters in the graph.
• Uses modularity as a proxy for embedding quality: lower modularity indicates better mixing of languages and higher potential utility for downstream tasks.
• Employs random projection trees to efficiently compute k-nearest neighbors for large-scale embeddings.
• Validates the metric by correlating modularity scores with performance on downstream tasks such as document classification and bilingual lexical induction.
• Uses modularity as a validation metric during model training to select better mapping matrices in unsupervised models like MUSE.

Experimental results

Research questions

• RQ1 Can modularity serve as a reliable intrinsic evaluation metric for cross-lingual word embeddings without requiring external linguistic resources?
• RQ2 Does modularity correlate with downstream task performance across diverse language pairs, including low-resource and distant language pairs?
• RQ3 How does modularity compare in predictive power to existing intrinsic metrics like QVEC-CCA and CSLS?
• RQ4 Can modularity be used effectively as a validation metric to improve unsupervised cross-lingual embedding models?
• RQ5 Does modularity capture unique information not captured by other intrinsic metrics?

Key findings

• Modularity shows a moderate to strong negative correlation with downstream task performance, with R² values up to 0.797 in cross-lingual document classification.
• Using modularity as a validation metric improves MUSE's performance on distant language pairs, with a +18.00% increase in P@1 for English-Japanese and +5.74% for English-Chinese.
• Ablation studies show that omitting modularity causes the largest drop in predictive accuracy (R² = 0.814 when all features are used), indicating it captures unique, complementary information.
• The optimal number of nearest neighbors for modularity computation is k = 3, as higher k values reduce correlation with downstream performance.
• Modularity outperforms other intrinsic metrics in predicting downstream performance, especially in low-resource settings where external resources are unavailable.
• The metric is effective even when applied to embeddings trained with no parallel data, demonstrating its utility in truly unsupervised settings.