[Paper Review] Robust Speech Recognition via Large-Scale Weak Supervision

Alec Radford, Jong Wook Kim+4 more|arXiv (Cornell University)|Dec 6, 2022

Speech Recognition and SynthesisComputer Science1,135 citations

TL;DR

Whisper trains on 680k hours of weakly supervised, multilingual data to achieve zero-shot robust speech recognition that rivals supervised models and approaches human performance in robustness without fine-tuning.

ABSTRACT

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

Motivation & Objective

• Explore whether large-scale weakly supervised pre-training can yield robust, zero-shot speech recognition across languages and tasks.
• Assess cross-domain robustness and zero-shot transfer on standard benchmarks without decoder fine-tuning.
• Investigate multilingual and multitask training benefits and compare to human performance under distribution shifts.
• Release models and inference code to establish a foundation for robust speech processing.

Proposed method

• Use an encoder-decoder Transformer trained on 680,000 hours of audio with transcripts spanning 96 languages and translation data.
• Represent multiple tasks (transcription, translation, voice activity detection, language identification) as a single sequence-to-sequence objective with special tokens.
• Train with minimal pre-processing, filtering to reduce machine-generated transcripts, and segment audio into 30-second chunks for training.
• Evaluate in zero-shot settings across diverse datasets to measure out-of-distribution robustness.
• Incrementally improve training with regularization (specAugment, stochastic depth, BPE dropout) in a later Large model variant.

Experimental results

Research questions

• RQ1 Can large-scale weakly supervised pre-training yield robust, zero-shot speech recognition across languages and tasks without decoder fine-tuning?
• RQ2 How does multilingual/multitask pre-training affect cross-domain robustness and zero-shot performance compared to supervised baselines?
• RQ3 To what extent do zero-shot Whisper models approach human performance in robustness on diverse datasets?
• RQ4 What is the impact of data quality filters and de-duplication on model performance and generalization?
• RQ5 How does Whisper perform in long-form transcription and under noisy conditions compared to specialized models?

Key findings

• Zero-shot Whisper models achieve competitive or superior robustness across multiple datasets relative to supervised models trained on LibriSpeech when evaluated out-of-distribution.
• On average, zero-shot Whisper reduces errors by about 55.2% on non-LibriSpeech datasets compared to a LibriSpeech-trained baseline with similar LibriSpeech performance.
• Whisper matches or approaches human robustness on several benchmarks, particularly under distribution shifts and noisy conditions.
• Multilingual and multitask pre-training show strong correlation between language data amount and zero-shot performance, with language data quantity predicting translation and recognition performance variably across tasks.
• Whisper demonstrates strong long-form transcription capabilities with competitive performance against commercial and open-source systems in many datasets.