[Paper Review] Do Explicit Alignments Robustly Improve Multilingual Encoders?

Shijie Wu, Mark Dredze|arXiv (Cornell University)|Oct 6, 2020

Natural Language Processing Techniques41 references4 citations

TL;DR

This paper proposes a contrastive learning-based alignment objective to improve multilingual encoders using noisy, low-resource bitexts like OPUS-100, rather than high-quality parallel data. Despite outperforming prior alignment methods on noisy data, extensive evaluation across multiple tasks, languages, and random seeds shows no significant improvement over strong baselines, especially when using larger models like XLM-RoBERTa-large, indicating that explicit alignment objectives offer limited gains under robust evaluation settings.

ABSTRACT

Multilingual BERT (mBERT), XLM-RoBERTa (XLMR) and other unsupervised multilingual encoders can effectively learn cross-lingual representation. Explicit alignment objectives based on bitexts like Europarl or MultiUN have been shown to further improve these representations. However, word-level alignments are often suboptimal and such bitexts are unavailable for many languages. In this paper, we propose a new contrastive alignment objective that can better utilize such signal, and examine whether these previous alignment methods can be adapted to noisier sources of aligned data: a randomly sampled 1 million pair subset of the OPUS collection. Additionally, rather than report results on a single dataset with a single model run, we report the mean and standard derivation of multiple runs with different seeds, on four datasets and tasks. Our more extensive analysis finds that, while our new objective outperforms previous work, overall these methods do not improve performance with a more robust evaluation framework. Furthermore, the gains from using a better underlying model eclipse any benefits from alignment training. These negative results dictate more care in evaluating these methods and suggest limitations in applying explicit alignment objectives.

Motivation & Objective

• To investigate whether explicit alignment objectives improve multilingual representation learning when trained on noisy, low-resource bitexts instead of high-quality parallel data.
• To develop a new contrastive learning-based alignment objective that better utilizes suboptimal word alignments from unsupervised aligners.
• To evaluate alignment methods under a more rigorous framework, including multiple random seeds, multiple languages, and multiple downstream tasks.
• To assess whether gains from alignment training are eclipsed by improvements from larger model architectures.

Proposed method

• Proposes a contrastive learning objective that optimizes for similarity between aligned sentence pairs from multilingual bitexts, using negative sampling to improve alignment robustness.
• Applies the contrastive objective to both high-quality bitexts (e.g., Europarl) and noisier, low-resource bitexts (OPUS-100), a 1M-sample subset of the OPUS collection.
• Uses unsupervised word alignment to generate weak alignment signals, then applies a contrastive loss that encourages aligned word pairs to have similar representations.
• Implements two variants: 'Weak Align' and 'Strong Align', differing in how negative samples are constructed and how the contrastive objective is optimized.
• Employs a memory-efficient gradient descent approach to learn orthogonal linear mappings between multilingual representations, avoiding full matrix storage.
• Evaluates models across four diverse NLP tasks (XNLI, POS, parsing, NER) and multiple languages using mean and standard deviation over multiple random seeds.

Experimental results

Research questions

• RQ1 Does a contrastive alignment objective improve multilingual encoder performance when trained on noisy, low-resource bitexts like OPUS-100, rather than high-quality bitexts?
• RQ2 Do explicit alignment objectives yield consistent and significant gains across multiple languages and tasks under a robust evaluation framework with multiple random seeds?
• RQ3 Can the performance gains from alignment training be attributed to the alignment objective itself, or are they overshadowed by model scale?
• RQ4 How do different alignment objectives (linear mapping, L2, contrastive) compare in terms of robustness and effectiveness on noisy data?
• RQ5 Does the performance of alignment-based methods degrade or disappear when evaluated with standard deviation across multiple runs, compared to single-seed evaluations?

Key findings

• The proposed contrastive alignment objective outperforms prior methods (e.g., Cao et al., 2020) on both high-quality and noisy bitexts, including OPUS-100.
• When evaluated with multiple random seeds and standard deviation, previously reported gains from alignment objectives on XNLI disappear, indicating low reproducibility under robust evaluation.
• Even with the improved contrastive objective, no significant performance gains are observed across four diverse tasks and multiple languages when using mBERT or XLM-RoBERTa-base.
• XLM-RoBERTa-large consistently outperforms all alignment-trained models, suggesting that model scale dominates any benefits from explicit alignment.
• The study concludes that explicit alignment objectives do not robustly improve multilingual encoders, especially when evaluated with proper statistical rigor and in the presence of larger models.