[Paper Review] SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

Paul-Ambroise Duquenne, Holger Schwenk+1 more|arXiv (Cornell University)|Aug 22, 2023

Natural Language Processing Techniques5 citations

TL;DR

Sonar introduces a multilingual, multimodal sentence embedding space trained via an encoder-decoder framework with joint translation, auto-encoding, and denoising objectives, achieving state-of-the-art performance on multilingual similarity search and zero-shot speech-to-text translation. It enables cross-lingual and cross-modal retrieval using a single text encoder for 200 languages and language-specific speech encoders, with open-sourced models available on GitHub.

ABSTRACT

We introduce SONAR, a new multilingual and multimodal fixed-size sentence embedding space. Our single text encoder, covering 200 languages, substantially outperforms existing sentence embeddings such as LASER3 and LabSE on the xsim and xsim++ multilingual similarity search tasks. Speech segments can be embedded in the same SONAR embedding space using language-specific speech encoders trained in a teacher-student setting on speech transcription data. Our encoders outperform existing speech encoders on similarity search tasks. We also provide a text decoder for 200 languages, which allows us to perform text-to-text and speech-to-text machine translation, including for zero-shot language and modality combinations. Our text-to-text results are competitive compared to the state-of-the-art NLLB~1B model, despite the fixed-size bottleneck representation. Our zero-shot speech-to-text translation results compare favorably with strong supervised baselines such as Whisper.

Motivation & Objective

• To develop a unified, language-agnostic sentence embedding space that supports both text and speech modalities across 200 languages.
• To improve multilingual sentence representation learning by combining translation, auto-encoding, and denoising objectives in a single framework.
• To enable zero-shot speech-to-text translation using a teacher-student training setup with multilingual text encoders as teachers.
• To evaluate the effectiveness of different training objectives on semantic similarity and decoding performance in multilingual and multimodal settings.
• To provide open-source text and speech encoders, along with a multilingual text decoder, for broad research and application use.

Proposed method

• The Sonar framework uses a multilingual encoder-decoder architecture initialized from the NLLB 1B model to learn fixed-size sentence embeddings.
• It jointly optimizes for translation, auto-encoding, and denoising objectives, with a cross-lingual similarity loss to enhance language-agnostic representations.
• Speech encoders for 37 languages are trained via a teacher-student distillation process, using the multilingual text encoder as the teacher and ASR transcriptions as supervision.
• The framework leverages a shared embedding space across modalities, enabling cross-modal similarity search and zero-shot translation.
• The text decoder is trained end-to-end to support zero-shot text-to-text and speech-to-text translation across language pairs.
• All models, including text and speech encoders and the text decoder, are released on GitHub for public use.

Experimental results

Research questions

• RQ1 How do different training objectives—translation, auto-encoding, denoising, and cross-lingual similarity—impact the quality of multilingual sentence embeddings in a fixed-size bottleneck?
• RQ2 Can a single multilingual text encoder trained with joint objectives outperform existing state-of-the-art sentence embedding models like Laser3 and LabSE on multilingual similarity search tasks?
• RQ3 To what extent can speech encoders trained via teacher-student distillation from a multilingual text encoder achieve high performance in cross-lingual and cross-modal similarity search?
• RQ4 How effective is the Sonar framework for zero-shot speech-to-text translation compared to supervised baselines like Whisper?
• RQ5 What is the impact of combining multiple objectives on the generalization and robustness of the learned sentence representations across diverse languages and modalities?

Key findings

• The Sonar text encoder outperforms existing models like Laser3 and LabSE on xsim and xsim++ multilingual similarity search benchmarks, demonstrating superior language-agnostic sentence representations.
• The model achieves competitive zero-shot text-to-text translation performance, matching the state-of-the-art NLLB 1B model despite using a fixed-size bottleneck.
• For zero-shot speech-to-text translation, Sonar achieves higher average BLEU scores than Whisper v2 large on the Fleurs test set, with particularly strong performance on Indian languages like Bengali, Hindi, and Tamil.
• The combination of translation and MSE losses significantly improves cross-lingual alignment, while excessive denoising degrades language-agnostic representation quality.
• Speech encoders trained via teacher-student distillation achieve strong performance on cross-modal similarity search, confirming the compatibility of text and speech embeddings in the same space.
• The Sonar framework enables effective zero-shot decoding across language and modality combinations, with open-sourced models available for research and deployment.