[Paper Review] Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

Yu Zhang, Wei Han+25 more|arXiv (Cornell University)|Mar 2, 2023

Speech Recognition and Synthesis112 citations

TL;DR

The Universal Speech Model (USM) is a large multilingual ASR system trained with vast unlabeled multilingual data and limited labeled data, achieving state-of-the-art results across 100+ languages for ASR and speech translation tasks, with strong YouTube captioning performance and robust long-form decoding.

ABSTRACT

We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.

Motivation & Objective

• Motivate building a universal ASR model that covers hundreds of languages and domains.
• Leverage large unlabeled multilingual audio and text data to pre-train a powerful encoder with a smaller labeled dataset for fine-tuning.
• Develop scalable training and fine-tuning pipelines enabling high performance across ASR and AST tasks.
• Demonstrate state-of-the-art results on benchmarks like SpeechStew, FLEURS, CORAAL, and CoVoST 2.

Proposed method

• Use a Conformer encoder backbone (600M and 2B parameter variants).
• Pre-train the encoder with BEST-RQ (random-projection quantizer) on 12M hours of unlabeled YouTube multilingual data (YT-NTL-U).
• Apply MOST (multi-objective supervised pre-training) to combine BEST-RQ with text-injection and supervised losses across unlabeled speech, unlabeled text, and paired speech-text data.
• Train supervised ASR targets (CTC and LAS) on labeled data (YT-SUP+ and Pub-S).
• Employ chunk-wise attention to enable robust long-form ASR at scale, with decoders (CTC, LAS, or RNN-T).
• Optionally attach residual adapters (≈2% of parameters) to frozen pre-trained encoders for language/domain adaptation.

Figure 1: An overview of our approach. Training is split into three stages. (i) The first stage trains a conformer backbone on a large unlabeled speech dataset, optimizing for the BEST-RQ objective. (ii) We continue training this speech representation learning model while optimizing for multiple obj

Experimental results

Research questions

• RQ1 Can a single large multilingual model achieve competitive or superior performance across ASR and AST for 100+ languages with limited labeled data?
• RQ2 How effective is multistage pre-training (BEST-RQ, MOST) combining unlabeled audio, unlabeled text, and paired data for downstream multilingual tasks?
• RQ3 Does chunk-wise attention enable robust long-form ASR without hallucination compared to segment-based decoding?
• RQ4 What is the impact of adapter-based fine-tuning versus full-model fine-tuning for many languages?

Key findings

• USM establishes new state-of-the-art results on multiple benchmarks: SpeechStew, FLEURS, CORAAL, and CoVoST 2 across various languages and domains.
• For YouTube captioning (73 languages), USM achieves under 30% WER on the designated setup.
• A 2B-parameter USM model trained with 90k hours of supervised data can outperform Whisper (trained on >400k hours) on selected languages and tasks.
• MOST (BEST-RQ + text-injection) enables strong adaptation to new domains; adapters add only about 2% of parameters to the frozen encoder and still deliver competitive results.
• Chunk-wise attention mitigates long-form ASR degradation, producing robust transcripts for long utterances.

Figure 2: (Left) † WERs (%) Our language expansion effort to support more languages on YouTube (73 languages) and extending to 100+ languages on the public dataset (FLEURS). Lower is better. To the best of our knowledge, no published model can successfully decode all 73 languages from our YouTube se