[Paper Review] A Common Semantic Space for Monolingual and Cross-Lingual Meta-Embeddings

Iker García-Ferrero, Rodrigo Agerri+1 more|arXiv (Cornell University)|Jan 17, 2020

Topic Modeling4 citations

TL;DR

This paper proposes a novel method for creating monolingual and cross-lingual meta-embeddings by projecting diverse word embeddings—derived from different sources, languages, and techniques—into a shared semantic space using linear transformations and averaging. The approach preserves original dimensionality, handles out-of-vocabulary words effectively, and achieves state-of-the-art performance on word similarity and POS tagging without hyperparameter tuning.

ABSTRACT

This paper presents a new technique for creating monolingual and cross-lingual meta-embeddings. Our method integrates multiple word embeddings created from complementary techniques, textual sources, knowledge bases and languages. Existing word vectors are projected to a common semantic space using linear transformations and averaging. With our method the resulting meta-embeddings maintain the dimensionality of the original embeddings without losing information while dealing with the out-of-vocabulary problem. An extensive empirical evaluation demonstrates the effectiveness of our technique with respect to previous work on various intrinsic and extrinsic multilingual evaluations, obtaining competitive results for Semantic Textual Similarity and state-of-the-art performance for word similarity and POS tagging (English and Spanish). The resulting cross-lingual meta-embeddings also exhibit excellent cross-lingual transfer learning capabilities. In other words, we can leverage pre-trained source embeddings from a resource-rich language in order to improve the word representations for under-resourced languages.

Motivation & Objective

• To develop a robust, scalable method for combining monolingual and cross-lingual word embeddings from diverse sources without information loss.
• To address the out-of-vocabulary (OOV) problem in meta-embedding construction, which often degrades performance when ignored.
• To eliminate the need for hyperparameter tuning across different embedding types, enabling consistent and efficient integration.
• To improve performance on both intrinsic (e.g., word similarity) and extrinsic (e.g., POS tagging) tasks using unified meta-embeddings.
• To enable effective cross-lingual transfer learning by leveraging high-quality embeddings from resource-rich languages to improve representations in low-resource languages.

Proposed method

• Project multiple source word embeddings into a common semantic space using VecMap, a state-of-the-art vector space mapping technique.
• Apply linear transformations to align embeddings from different sources, languages, and models into a shared vector space.
• Use averaging of aligned embeddings to generate final meta-embeddings, preserving the original dimensionality and minimizing information loss.
• Introduce a neural network-based OOV treatment strategy that assigns meaningful representations to words missing in certain source embeddings.
• Maintain a single, fixed hyperparameter across all experiments, ensuring methodological simplicity and robustness.
• Combine embeddings from diverse sources—including monolingual and cross-lingual models, knowledge bases, and multiple languages—into a unified meta-embedding.

Experimental results

Research questions

• RQ1 Can a common semantic space be effectively constructed for heterogeneous word embeddings from different languages, sources, and models?
• RQ2 How does proper handling of out-of-vocabulary (OOV) words impact the quality of meta-embeddings in both monolingual and cross-lingual settings?
• RQ3 To what extent can a single, non-tuned hyperparameter strategy outperform existing meta-embedding methods requiring extensive tuning?
• RQ4 Can cross-lingual meta-embeddings improve performance on monolingual downstream tasks like POS tagging and word similarity?
• RQ5 Does the proposed method achieve state-of-the-art results on intrinsic and extrinsic evaluations compared to prior linear and neural meta-embedding approaches?

Key findings

• The proposed method achieves new state-of-the-art performance on the Word Similarity task, outperforming previous methods including supervised neural models.
• For POS tagging on the Universal Dependencies 1.2 dataset, the method sets new SOTA results with 96.96% accuracy in English and 97.35% in Spanish.
• The cross-lingual meta-embeddings demonstrate strong transfer learning capabilities, improving representations in low-resource languages using high-quality source embeddings.
• The method outperforms concatenation-based baselines in terms of performance-to-dimensionality ratio, avoiding the high dimensionality of concatenation while minimizing information loss.
• The OOV treatment strategy significantly improves performance, as zero-vector assignment for OOV words leads to large performance drops.
• The method is robust and efficient, requiring no hyperparameter tuning across diverse embedding types, enabling rapid integration of large numbers of embeddings.