[Paper Review] Multilingual Universal Sentence Encoder for Semantic Retrieval

Yinfei Yang, Daniel Cer+10 more|arXiv (Cornell University)|Jul 9, 2019

Topic Modeling13 references67 citations

TL;DR

The paper presents two multilingual sentence encoding models (Transformer and CNN) that map 16 languages into a shared semantic space using a multi-task dual-encoder, achieving competitive results on semantic retrieval, bitext retrieval, and retrieval-based QA, and released on TensorFlow Hub.

ABSTRACT

We introduce two pre-trained retrieval focused multilingual sentence encoding models, respectively based on the Transformer and CNN model architectures. The models embed text from 16 languages into a single semantic space using a multi-task trained dual-encoder that learns tied representations using translation based bridge tasks (Chidambaram al., 2018). The models provide performance that is competitive with the state-of-the-art on: semantic retrieval (SR), translation pair bitext retrieval (BR) and retrieval question answering (ReQA). On English transfer learning tasks, our sentence-level embeddings approach, and in some cases exceed, the performance of monolingual, English only, sentence embedding models. Our models are made available for download on TensorFlow Hub.

Motivation & Objective

• Enable multilingual semantic similarity by embedding 16 languages in a single semantic space.
• Develop retrieval-focused encoders using a multi-task dual-encoder framework with bridge tasks.
• Provide efficient, transferable sentence embeddings suitable for SR, BR, and ReQA tasks across languages.
• Assess transfer performance to English tasks and cross-lingual retrieval scenarios.

Proposed method

• Two multilingual encoders (Transformer and CNN) map sentences to a shared space.
• Multi-task dual-encoder training with question-answer prediction, translation ranking, and natural language inference tasks.
• SentencePiece subword tokenization with a shared 128k vocabulary across 16 languages.
• Training data from QA pairs, translation pairs, SNLI, MultiNLI, and translated data to balance languages.
• CNN and Transformer encoders with specific hyperparameters; QA variant USE QA Trans+Cxt for context-aware retrieval.

Experimental results

Research questions

• RQ1 How well can a multilingual dual-encoder embed 16 languages into a unified semantic space for retrieval tasks?
• RQ2 What is the performance of Transformer vs CNN architectures on semantic retrieval, bitext retrieval, and ReQA in a multilingual setting?
• RQ3 Can cross-lingual retrieval approaches reach close to monolingual performance across language pairs?
• RQ4 How does transfer learning to English tasks compare with monolingual English models?
• RQ5 What are the resource trade-offs (speed, memory) between CNN and Transformer encoders in multilingual settings?

Key findings

• The multilingual Transformer and CNN encoders achieve competitive performance on SR, BR, and ReQA tasks compared to state-of-the-art baselines.
• The models support cross-lingual semantic retrieval and cross-lingual ReQA with results approaching monolingual performance for many languages.
• SentencePiece enables broad language coverage with high character coverage across the 16 languages; English transfer tasks show competitive results to monolingual models.
• Transformer tends to outperform CNN in most tasks, though CNN offers lower memory footprint and faster inference, especially on longer texts.
• The models are publicly available via TensorFlow Hub with documentation and Colab notebooks.