[Paper Review] Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition

Zhang Yu, James Qin+6 more|arXiv (Cornell University)|Oct 20, 2020

Speech Recognition and SynthesisComputer Science53 references200 citations

TL;DR

The paper combines wav2vec 2.0 pre-training with noisy student training and SpecAugment to push state-of-the-art WER on LibriSpeech using Libri-Light unlabeled data, achieving 1.4%/2.6% on test/test-other with large Conformer models.

ABSTRACT

We employ a combination of recent developments in semi-supervised learning for automatic speech recognition to obtain state-of-the-art results on LibriSpeech utilizing the unlabeled audio of the Libri-Light dataset. More precisely, we carry out noisy student training with SpecAugment using giant Conformer models pre-trained using wav2vec 2.0 pre-training. By doing so, we are able to achieve word-error-rates (WERs) 1.4%/2.6% on the LibriSpeech test/test-other sets against the current state-of-the-art WERs 1.7%/3.3%.

Motivation & Objective

• Motivate the use of large-scale semi-supervised learning for ASR to improve performance on LibriSpeech.
• Demonstrate that combining pre-training with iterative self-training yields state-of-the-art WER.
• Explore how model size and pre-training interact to enhance ASR performance.

Proposed method

• Use Conformer-based sequence-transducer models with LSTM decoders.
• Pre-train encoders with wav2vec 2.0 style self-supervised pre-training using Libri-Light unlabeled data (log-mel inputs; masking; contrastive loss; replace quantization with a linear projection).
• Apply Noisy Student Training with adaptive SpecAugment in a multi-generation NST loop where teacher labels generated on unlabeled data augment supervised data.
• Fine-tune pre-trained checkpoints in NST with SpecAugment, LM fusion, and batch-wise data mixing (1:9 supervised:teacher-labeled).
• Experiment with Conformer XL/XXL/XXL+ sizes to study the effect of pre-training versus training from scratch.
• Evaluate on LibriSpeech dev, dev-clean/dev-other, test, test-other with and without LM fusion.

Experimental results

Research questions

• RQ1 Can combining wav2vec 2.0 pre-training with iterative self-training (NST) and SpecAugment push LibriSpeech WER beyond prior SSL methods?
• RQ2 How does increasing model capacity interact with pre-training and SSL to improve ASR performance?
• RQ3 What is the impact of data mixing, LM fusion, and ablations in NST on final WERs?
• RQ4 Is relying on large unlabeled data from Libri-Light essential for gains, and how sensitive are results to pre-training input representations (log-mel vs waveform)?

Key findings

• The NST+pre-training pipeline with giant Conformers achieves state-of-the-art WERs on LibriSpeech dev/test and dev-other/test-other (e.g., 1.3%/2.7%/1.5%/2.8% without LM and 1.3%/2.7%/1.5%/2.7% with LM for Gen3 Conformer XXL).
• Model size alone does not guarantee gains; gains materialize when SSL methods are applied (pre-training and NST).
• Gen3 Conformer XXL and XXL+ achieve the best results, with XXL+ providing modest further gains over XXL under the NST framework.
• Pre-training with log-mel inputs and a linear projection block (without quantization) can still yield strong results, suggesting the projection head design can be varied.
• Ablations show that maximizing the amount of teacher-generated data (vs filtering/balancing) benefits generation-1 performance in this setup.
• Scaling pre-training data and model size up to 1B parameters yields consistent improvements, unlike supervised-only scaling.