[Paper Review] Synt++: Utilizing Imperfect Synthetic Data to Improve Speech Recognition

Ting-Yao Hu, Mohammadreza Armandpour+4 more|arXiv (Cornell University)|Oct 21, 2021

Speech Recognition and Synthesis4 citations

TL;DR

Synt++ improves speech recognition by addressing distribution gaps between real and synthetic speech data using two key techniques: rejection sampling to filter out low-quality synthetic samples and separate batch normalization statistics for real and synthetic data during training. This approach reduces WER by up to 13% and false accept rate by up to 18% compared to naive synthetic data usage.

ABSTRACT

With recent advances in speech synthesis, synthetic data is becoming a viable alternative to real data for training speech recognition models. However, machine learning with synthetic data is not trivial due to the gap between the synthetic and the real data distributions. Synthetic datasets may contain artifacts that do not exist in real data such as structured noise, content errors, or unrealistic speaking styles. Moreover, the synthesis process may introduce a bias due to uneven sampling of the data manifold. We propose two novel techniques during training to mitigate the problems due to the distribution gap: (i) a rejection sampling algorithm and (ii) using separate batch normalization statistics for the real and the synthetic samples. We show that these methods significantly improve the training of speech recognition models using synthetic data. We evaluate the proposed approach on keyword detection and Automatic Speech Recognition (ASR) tasks, and observe up to 18% and 13% relative error reduction, respectively, compared to naively using the synthetic data.

Motivation & Objective

• To address the distribution gap between real and synthetic speech data in speech recognition training.
• To mitigate artifacts, over/under-sampling, and domain shift caused by imperfect synthetic data.
• To improve ASR and keyword detection performance using synthetic data without requiring perfect synthesis.
• To develop a practical framework that enhances model robustness when training with imperfect synthetic data.

Proposed method

• A rejection sampling algorithm is used to filter synthetic samples based on a trained discriminator's confidence in classifying them as real or synthetic.
• The discriminator is trained to distinguish between real and synthetic speech samples, and its output probability guides stochastic acceptance or rejection of synthetic samples.
• Separate batch normalization (BN) statistics are computed and maintained for real and synthetic data during training.
• During inference, only real data BN statistics are used to prevent domain shift from synthetic data statistics.
• The method is applied end-to-end in ASR and keyword detection models, with no architectural changes required.
• The approach is evaluated on LibriSpeech and Speech Commands datasets using standard metrics like WER and false accept rate.

Experimental results

Research questions

• RQ1 Can rejection sampling reduce the impact of artifacts and distributional mismatches in synthetic speech data on ASR performance?
• RQ2 Does using separate batch normalization statistics for real and synthetic data improve model generalization and reduce error rates?
• RQ3 To what extent can synthetic data improve ASR and keyword detection when combined with Synt++ techniques compared to naive synthetic data usage?
• RQ4 How does Synt++ perform across different model sizes and data scales in low-resource and full-resource settings?

Key findings

• Synt++ achieved a relative WER reduction of up to 13% on the LibriSpeech 960h dataset compared to training with real and synthetic data naively.
• On the LibriSpeech 100h dataset, Synt++ reduced WER by 18% relative to the baseline using real and synthetic data.
• In keyword detection, Synt++ reduced the average false accept rate by up to 18% compared to using synthetic data without filtering.
• The ablation study confirmed that both rejection sampling and dual batch normalization contribute significantly to performance gains.
• The method achieved consistent improvements across different model sizes (22M and 116M parameters) and data settings.
• The results demonstrate that Synt++ effectively mitigates distribution shift and enables reliable use of imperfect synthetic data in speech recognition.