[Paper Review] SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network

William Chan, Daniel Park+4 more|arXiv (Cornell University)|Apr 5, 2021

Speech Recognition and SynthesisComputer Science54 references75 citations

TL;DR

SpeechStew mixes multiple public ASR datasets to train a single large end-to-end model, achieving state-of-the-art or near state-of-the-art results without an external language model and showing strong transfer learning to CHiME-6.

ABSTRACT

We present SpeechStew, a speech recognition model that is trained on a combination of various publicly available speech recognition datasets: AMI, Broadcast News, Common Voice, LibriSpeech, Switchboard/Fisher, Tedlium, and Wall Street Journal. SpeechStew simply mixes all of these datasets together, without any special re-weighting or re-balancing of the datasets. SpeechStew achieves SoTA or near SoTA results across a variety of tasks, without the use of an external language model. Our results include 9.0\% WER on AMI-IHM, 4.7\% WER on Switchboard, 8.3\% WER on CallHome, and 1.3\% on WSJ, which significantly outperforms prior work with strong external language models. We also demonstrate that SpeechStew learns powerful transfer learning representations. We fine-tune SpeechStew on a noisy low resource speech dataset, CHiME-6. We achieve 38.9\% WER without a language model, which compares to 38.6\% WER to a strong HMM baseline with a language model.

Motivation & Objective

• Motivate improving generalization in end-to-end ASR by leveraging diverse, publicly available datasets without re-weighting or domain-specific adjustments.
• Propose a simple multi-domain training recipe that scales a single model to 100M or 1B parameters using mixed data.
• Demonstrate transfer learning by fine-tuning the pre-trained model on a low-resource, noisy dataset (CHiME-6).
• Assess performance across a wide range of tasks without using an external language model during inference.

Proposed method

• Train a single large Conformer-RNN-T model (100M and 1B parameter configurations).
• Combine AMI, Common Voice, English Broadcast News, LibriSpeech, Switchboard/Fisher, TED-LIUM v3, and WSJ without data re-weighting.
• Pre-train the 1B model with wav2vec 2.0 on LibriLight before supervised SpeechStew training.
• Use SpecAugment and large batch training with Adam/Adafactor optimizers and exponential moving average for evaluation.
• Fine-tune the pretrained SpeechStew on CHiME-6 to demonstrate transfer learning capabilities.
• No external language model is used during inference.

Experimental results

Research questions

• RQ1 Can simply mixing multiple public speech recognition datasets train a single large model that achieves state-of-the-art or near state-of-the-art results across diverse ASR tasks without an external language model?
• RQ2 Do large multi-domain models preserve transfer learning capabilities that improve performance on low-resource, noisy datasets after fine-tuning?
• RQ3 How does wav2vec pre-training interact with large supervised SpeechStew models for very large parameter counts?

Key findings

• SpeechStew achieves 9.0 WER on AMI-IHM and 21.7 WER on Common Voice (100M model).
• SpeechStew achieves 12.1 WER (clean) and 9.7 WER (with punctuation normalization) on LibriSpeech.
• SpeechStew achieves 4.7 WER on Switchboard and 8.3 WER on CallHome.
• SpeechStew achieves 1.3 WER on WSJ.
• SpeechStew 1B with LibriLight pre-training improves some tasks over the 100M model and shows strong transfer to CHiME-6 (finetuned: 38.9 WER on CHiME-6 eval).
• Zero-shot CHiME-6 (SpeechStew 1B) yields 53.7 WER on eval with 39.2 on dev; finetuning improves to 38.9 WER on eval.