[Paper Review] SpecAugment on Large Scale Datasets

Daniel Park, Yu Zhang+6 more|arXiv (Cornell University)|Dec 11, 2019

Speech Recognition and Synthesis26 references4 citations

TL;DR

This paper demonstrates that SpecAugment, a simple spectrogram-level data augmentation technique, significantly improves end-to-end automatic speech recognition (ASR) performance on large-scale industrial datasets like Google's Multidomain Dataset and LibriSpeech 960h. By combining SpecAugment with multistyle training (MTR) data via data mixing, the authors achieve state-of-the-art results—2.2% WER on LibriSpeech test-clean and 5.2% on test-other—while maintaining low computational cost and scalability.

ABSTRACT

Recently, SpecAugment, an augmentation scheme for automatic speech recognition that acts directly on the spectrogram of input utterances, has shown to be highly effective in enhancing the performance of end-to-end networks on public datasets. In this paper, we demonstrate its effectiveness on tasks with large scale datasets by investigating its application to the Google Multidomain Dataset (Narayanan et al., 2018). We achieve improvement across all test domains by mixing raw training data augmented with SpecAugment and noise-perturbed training data when training the acoustic model. We also introduce a modification of SpecAugment that adapts the time mask size and/or multiplicity depending on the length of the utterance, which can potentially benefit large scale tasks. By using adaptive masking, we are able to further improve the performance of the Listen, Attend and Spell model on LibriSpeech to 2.2% WER on test-clean and 5.2% WER on test-other.

Motivation & Objective

• To evaluate the effectiveness of SpecAugment on large-scale, multi-domain ASR datasets beyond public benchmarks.
• To investigate whether SpecAugment can outperform or complement more complex data augmentation techniques like Multistyle Training (MTR) in industrial-scale settings.
• To explore adaptive time masking policies that adjust to utterance length for improved generalization on diverse-length inputs.
• To demonstrate that SpecAugment enhances streaming ASR models, including RNN-T architectures, despite training instability from layer normalization.

Proposed method

• Apply SpecAugment—comprising time warping, frequency masking, and time masking—directly to spectrograms of raw audio inputs.
• Use a fixed policy (SpecAugBasic) with two frequency masks (F=27) and two time masks (T=50), without time warping, on clean training data.
• Introduce adaptive time masking where the number and size of time masks vary based on utterance length to improve generalization on long and short sequences.
• Mix SpecAugmented clean data with MTR-augmented data at an 8:2 ratio to balance domain generalization and robustness.
• Stabilize training of streaming RNN-T models with time masks by injecting Gaussian noise into masked regions to counteract gradient instability from layer normalization.
• Train and evaluate on multiple test sets: natural domains (e.g., Search, Call, YouTube) and synthetic domains (e.g., Search-Noisy) created via MTR.

Experimental results

Research questions

• RQ1 Does SpecAugment maintain its performance gains on large-scale, multi-domain industrial datasets, such as Google's Multidomain Dataset?
• RQ2 How does SpecAugment compare to or complement Multistyle Training (MTR), a state-of-the-art data augmentation method for industrial ASR?
• RQ3 Can adaptive time masking—where mask parameters vary with utterance length—improve performance over fixed-specification SpecAugment on large-scale tasks?
• RQ4 Does SpecAugment provide measurable gains in streaming ASR models, particularly RNN-Ts, despite training instabilities from layer normalization?
• RQ5 What is the optimal strategy for combining SpecAugment with MTR data to maximize performance across diverse test domains?

Key findings

• SpecAugment applied to clean data outperforms the MTR baseline on all natural test sets (e.g., Search, Call, YouTube), achieving 6.2% WER on Search and 4.2% on Call.
• Applying SpecAugment on top of MTR-augmented data degrades performance across most domains, indicating negative interaction between the two methods.
• Mixing 80% SpecAugmented clean data with 20% MTR-augmented data yields the best overall performance, improving WER across all test sets compared to either method alone.
• On LibriSpeech 960h, the adaptive time masking variant of SpecAugment reduces WER to 2.2% on test-clean and 5.2% on test-other, surpassing the baseline performance reported in the original SpecAugment paper.
• Time masking alone provides significant gains in streaming RNN-T models, even when combined with layer normalization, especially when stabilized with injected noise in masked regions.
• While adaptive time masking did not outperform fixed policies on the Google Multidomain Dataset, it showed measurable gains on LibriSpeech, suggesting domain-specific benefits for adaptive strategies.