[Paper Review] Data Augmenting Contrastive Learning of Speech Representations in the Time Domain

Eugene Kharitonov, Morgane Rivière+5 more|arXiv (Cornell University)|Jul 2, 2020

Speech Recognition and Synthesis50 references92 citations

TL;DR

The paper introduces WavAugment, a time-domain data augmentation library for CPC-based speech representation learning, showing that past-only augmentation and independent past/future augmentations substantially improve performance, achieving 18-22% relative gains on Libri-light and competitive results on Zero Speech with larger data. It also demonstrates gains in low-resource phoneme classification.

ABSTRACT

Contrastive Predictive Coding (CPC), based on predicting future segments of speech based on past segments is emerging as a powerful algorithm for representation learning of speech signal. However, it still under-performs other methods on unsupervised evaluation benchmarks. Here, we introduce WavAugment, a time-domain data augmentation library and find that applying augmentation in the past is generally more efficient and yields better performances than other methods. We find that a combination of pitch modification, additive noise and reverberation substantially increase the performance of CPC (relative improvement of 18-22%), beating the reference Libri-light results with 600 times less data. Using an out-of-domain dataset, time-domain data augmentation can push CPC to be on par with the state of the art on the Zero Speech Benchmark 2017. We also show that time-domain data augmentation consistently improves downstream limited-supervision phoneme classification tasks by a factor of 12-15% relative.

Motivation & Objective

• Investigate the impact of time-domain data augmentations on CPC-based speech representation learning.
• Evaluate which augmentation strategies (past vs. past+future) yield the best improvements.
• Extend findings to French and Mandarin Zero Speech benchmarks and to limited-supervision phoneme classification.

Proposed method

• Use CPC2 architecture with an encoder, context network, and multi-head predictor to learn predictive representations of future speech segments.
• Introduce WavAugment, a time-domain augmentation library built on libsox for on-the-fly augmentations during data loading.
• Experiment with augmentations including pitch, additive noise, reverberation, band reject, and tdrop, applied to past and/or future segments in CPC training.
• Assess performance using Libri-light ABX metric for unsupervised representation learning across within- and across-speaker settings.
• Extend experiments to English, French, and Mandarin ZeroSpeech 2017 benchmarks with in-domain and out-of-domain training data.
• Evaluate semi-supervised improvements by fine-tuning with limited Libri-light labeled data on top of CPC features.

Experimental results

Research questions

• RQ1 How do time-domain augmentations affect CPC-based speech representation learning?
• RQ2 Which augmentation strategies (past-only vs. past+future) most improve unsupervised and semi-supervised speech tasks?
• RQ3 Can time-domain augmentation generalize across languages (English, French, Mandarin) and support limited supervision tasks?

Key findings

• Data augmentation yields 18-22% relative improvements over non-augmented CPC features on Libri-light.
• Pitch and additive noise are the most effective augmentations for CPC in English, Mandarin, and French.
• Past-only augmentation or independent past/future augmentations outperform alternatives for most augmentations; bandrej was detrimental and was dropped.
• CPC2+WavAug outperforms CPC2 on Libri-light across within- and across-speaker ABX tests in both clean and other conditions.
• With out-of-domain training data, CPC2+WavAug reaches or surpasses state-of-the-art baselines on ZeroSpeech 2017 in some settings but remains data-efficient only with larger training sets.
• Data augmentation enables substantial gains in semi-supervised phoneme classification (12-15% relative) and can beat large pretraining baselines with much smaller labeled-data budgets.
• Time-domain augmentations (pitch and noise) can outperform common spectral-domain augmentations (e.g., SpecAugment) in CPC training.