[Paper Review] Bitext Mining for Low-Resource Languages via Contrastive Learning

Weiting Tan, Philipp Koehn|arXiv (Cornell University)|Aug 23, 2022

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes fine-tuning sentence transformers with Multiple Negative Ranking Loss via contrastive learning to improve bitext mining for low-resource languages. The method achieves state-of-the-art results on Khmer and Pashto, boosting NMT BLEU scores by over 1 point compared to prior methods, demonstrating superior sentence alignment and filtering performance using improved cross-lingual sentence representations.

ABSTRACT

Mining high-quality bitexts for low-resource languages is challenging. This paper shows that sentence representation of language models fine-tuned with multiple negatives ranking loss, a contrastive objective, helps retrieve clean bitexts. Experiments show that parallel data mined from our approach substantially outperform the previous state-of-the-art method on low resource languages Khmer and Pashto.

Motivation & Objective

• Improve bitext mining quality for low-resource languages such as Khmer and Pashto, where parallel data is scarce.
• Address limitations in existing sentence alignment and filtering pipelines that rely on heuristic or suboptimal sentence representations.
• Develop a robust, scalable method for aligning and filtering sentence pairs using contrastive learning on cross-lingual sentence embeddings.
• Outperform existing state-of-the-art systems in the WMT 2020 parallel corpus filtering task for low-resource language pairs.
• Provide a publicly available toolkit for replicating and extending the method to other language pairs.

Proposed method

• Fine-tune a sentence-BERT (SBERT) model using Multiple Negative Ranking (MNR) loss to learn improved cross-lingual sentence representations.
• Construct positive pairs from aligned sentence pairs and negative pairs using neighboring sentences (window size W) and random samples (R) from the target side.
• Optimize the MNR loss to maximize the cosine similarity between positive pairs while minimizing similarity to negative pairs.
• Use the fine-tuned SBERT embeddings as a scoring function in Vecalign for sentence alignment and in a margin-based filtering mechanism for noisy pair removal.
• Apply Faiss for efficient k-nearest neighbor search during filtering, using a ratio-based margin function to rank sentence pairs.
• Train and evaluate the system on document-aligned corpora from ParaCrawl, using WMT 2020 evaluation protocols.

Experimental results

Research questions

• RQ1 Can contrastive fine-tuning of sentence transformers improve bitext mining quality for low-resource languages like Khmer and Pashto?
• RQ2 How does MNR-fine-tuned SBERT compare to LASER and other state-of-the-art sentence representations in sentence alignment and filtering tasks?
• RQ3 Does using a contrastive objective lead to better generalization and robustness when mining parallel data from noisy, web-crawled monolingual corpora?
• RQ4 To what extent does the choice of sentence representation impact downstream NMT performance in low-resource settings?
• RQ5 Can a single fine-tuned model effectively serve both alignment and filtering tasks with improved efficiency and accuracy?

Key findings

• The SBERT model fine-tuned with MNR loss achieved a BLEU score of 11.83 on Khmer and 10.43 on Pashto, outperforming the previous SOTA system by over 1 BLEU point.
• SBERT-ALIGN combined with SBERT-FILTER achieved the highest BLEU score across all methods, demonstrating the effectiveness of the fine-tuned representation in both alignment and filtering.
• The method outperformed LASER-based alignment and filtering by approximately 2 BLEU points, showing a significant improvement in NMT performance.
• The HUAWEI-FILTER method performed best among filtering techniques, but SBERT-based alignment and filtering still surpassed LASER-based counterparts by a large margin.
• The fine-tuned SBERT model showed superior ability to distinguish correct sentence pairs from noisy ones, as evidenced by higher similarity scores on correct pairs and lower scores on incorrect ones, especially in challenging cases like ambiguous or out-of-vocabulary terms.
• The approach is scalable and efficient, enabling fast inference on millions of sentence pairs using pre-computed embeddings and Faiss-based nearest neighbor search.