[Paper Review] Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders

Ivan Vulić, Goran Glavašš+4 more|arXiv (Cornell University)|Apr 30, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a simple contrastive fine-tuning method to extract and expose hidden cross-lingual lexical knowledge from multilingual sentence encoders (SEs), achieving state-of-the-art performance on cross-lingual lexical tasks like bilingual lexical induction (BLI), with gains of up to +10 Precision@1 points using only 1k–5k word translation pairs. The approach effectively 'rewires' SEs into powerful lexical encoders without requiring large-scale parallel data.

ABSTRACT

Pretrained multilingual language models (LMs) can be successfully transformed into multilingual sentence encoders (SEs; e.g., LaBSE, xMPNet) via additional fine-tuning or model distillation with parallel data. However, it remains unclear how to best leverage them to represent sub-sentence lexical items (i.e., words and phrases) in cross-lingual lexical tasks. In this work, we probe SEs for the amount of cross-lingual lexical knowledge stored in their parameters, and compare them against the original multilingual LMs. We also devise a simple yet efficient method for exposing the cross-lingual lexical knowledge by means of additional fine-tuning through inexpensive contrastive learning that requires only a small amount of word translation pairs. Using bilingual lexical induction (BLI), cross-lingual lexical semantic similarity, and cross-lingual entity linking as lexical probing tasks, we report substantial gains on standard benchmarks (e.g., +10 Precision@1 points in BLI). The results indicate that the SEs such as LaBSE can be 'rewired' into effective cross-lingual lexical encoders via the contrastive learning procedure, and that they contain more cross-lingual lexical knowledge than what 'meets the eye' when they are used as off-the-shelf SEs. This way, we also provide an effective tool for harnessing 'covert' multilingual lexical knowledge hidden in multilingual sentence encoders.

Motivation & Objective

• To investigate whether multilingual sentence encoders (SEs) store significant cross-lingual lexical knowledge beyond their off-the-shelf performance.
• To address the gap where SEs underperform compared to static cross-lingual word embeddings (CLWEs) in sub-sentence lexical tasks like BLI.
• To develop a lightweight, efficient method to expose this latent lexical knowledge via contrastive learning using only small bilingual dictionaries.
• To demonstrate that the resulting specialized encoders outperform both original multilingual LMs and SEs on standard cross-lingual lexical probing tasks.
• To validate the effectiveness of the method on low-resource language pairs and explore fusion with static CLWEs for further gains.

Proposed method

• Fine-tune multilingual sentence encoders (e.g., LaBSE, xMPNET) and multilingual LMs (e.g., mBERT, XLM-R) using a contrastive learning objective on small bilingual lexicons (1k–5k word pairs).
• Use a contrastive loss that pulls together the dense vector representations of word translations while pushing apart non-translations, optimizing for cross-lingual lexical alignment.
• Apply the fine-tuning procedure independently per language pair, enabling dedicated lexical specialization for each pair.
• Use isolated word or phrase inputs to the encoder, avoiding context dependency to isolate lexical knowledge in the model parameters.
• Interpolate the resulting cross-lingual lexical vectors with static CLWEs (e.g., from Artetxe et al.) to further boost performance.
• Evaluate the method on three standard tasks: bilingual lexical induction (BLI), cross-lingual lexical semantic similarity (XLSIM), and cross-lingual entity linking (XL-EL).

Experimental results

Research questions

• RQ1 Do multilingual sentence encoders contain more cross-lingual lexical knowledge than currently observable when used off-the-shelf?
• RQ2 Can a lightweight contrastive fine-tuning procedure with minimal parallel data (1k–5k word pairs) effectively expose this hidden lexical knowledge?
• RQ3 How does the performance of the resulting specialized lexical encoders compare to both original multilingual LMs and SEs on standard cross-lingual lexical tasks?
• RQ4 Does the method yield particularly strong gains for low-resource language pairs, where data is scarce?
• RQ5 Can combining the fine-tuned encoders with static CLWEs further improve performance on cross-lingual lexical tasks?

Key findings

• The proposed contrastive fine-tuning procedure yields up to +10 Precision@1 points improvement on standard BLI benchmarks, significantly outperforming both original multilingual LMs and SEs.
• Multilingual SEs like LaBSE and xMPNET contain substantially more cross-lingual lexical knowledge than their off-the-shelf counterparts, which is not fully utilized without additional fine-tuning.
• The method achieves state-of-the-art performance on BLI, XLSIM, and XL-EL tasks using only 1k–5k word translation pairs, making it highly efficient and scalable.
• Particularly large performance gains are observed on low-resource language pairs, demonstrating the method’s effectiveness in low-data regimes.
• Interpolating the fine-tuned lexical vectors with static CLWEs leads to even stronger performance, suggesting complementary strengths between learned and static embeddings.
• The approach is model-agnostic and can be applied to any multilingual encoder, offering a general-purpose tool for unlocking latent lexical knowledge.