[Paper Review] Language-agnostic BERT Sentence Embedding

Fangxiaoyu Feng, Yinfei Yang+3 more|arXiv (Cornell University)|Jul 3, 2020

Topic Modeling195 citations

TL;DR

LaBSE introduces a multilingual sentence embedding model built on pre-trained language models and dual-encoders, achieving state-of-the-art cross-lingual retrieval across 109+ languages with strong performance on bi-text mining and competitive transfer results.

ABSTRACT

While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning (Reimers and Gurevych, 2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) (Conneau and Lample, 2019), dual encoder translation ranking (Guo et al., 2018), and additive margin softmax (Yang et al., 2019a). We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks (Conneau and Kiela, 2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh and en-de. We publicly release our best multilingual sentence embedding model for 109+ languages at https://tfhub.dev/google/LaBSE.

Motivation & Objective

• Motivate multilingual sentence embeddings for clustering, retrieval, and downstream transfer tasks.
• Investigate coupling pre-trained language models with dual-encoder training for cross-lingual translation ranking.
• Assess the impact of pre-training, negative sampling, vocabulary choices, data quality, and data quantity on multilingual embeddings.
• Demonstrate scalability to 109+ languages and public release of the model for broad use.

Proposed method

• Use dual-encoder architectures with shared BERT-based encoders to map sentences into a common embedding space.
• Pre-train with Masked Language Modeling (MLM) and Translation Language Modeling (TLM) on monolingual and bilingual data.
• Fine-tune with translation ranking losses, including additive margin softmax, to encourage alignment of translations in the embedding space.
• Employ in-batch and cross-accelerator negative sampling to scale training with large batch sizes.
• Experiment with public mBERT vocabularies and a customized vocabulary to study impact on performance.
• Mine parallel data from CommonCrawl using LaBSE embeddings for NMT training as a demonstration of downstream data utility.

Experimental results

Research questions

• RQ1 Can a large multilingual encoder pre-trained on MLM/TLM produce high-quality sentence embeddings across many languages without language-specific tuning?
• RQ2 What is the impact of additive margin softmax and pre-training on cross-lingual retrieval and mining tasks?
• RQ3 How does LaBSE perform on bi-text retrieval and bitext mining compared to prior state-of-the-art models across diverse language sets?
• RQ4 To what extent can a single model cover 109+ languages and still deliver competitive downstream transfer performance?
• RQ5 What is the effect of vocabulary choice and data quantity on cross-lingual embedding quality?

Key findings

• LaBSE achieves state-of-the-art performance on bi-text retrieval and parallel text mining across multiple tasks and languages.
• The additive margin softmax significantly improves cross-lingual embedding performance across configurations.
• Pre-training dramatically reduces required parallel data (up to 80% reduction) while boosting performance.
• LaBSE delivers strong results on low-resource languages and languages without explicit training data, outperforming prior multilingual models like LASER and m-USE in several settings.
• On downstream transfer (SentEval), LaBSE remains competitive with English-centric and multilingual baselines, despite its broad language coverage.
• The model is publicly released for 109+ languages, and experiments show viable use in mining parallel data for NMT (en-zh, en-de).