[Paper Review] Text-To-Speech Data Augmentation for Low Resource Speech Recognition

Rodolfo Zevallos|arXiv (Cornell University)|Apr 1, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a novel text-to-speech (TTS)-based data augmentation method for low-resource, agglutinative languages like Quechua, combining sequence-to-sequence (seq2seq) text generation with TTS to synthesize both synthetic text and audio. The approach achieves an 8.73% relative reduction in word error rate (WER) on Quechua ASR models, demonstrating significant improvement in low-resource speech recognition performance.

ABSTRACT

Nowadays, the main problem of deep learning techniques used in the development of automatic speech recognition (ASR) models is the lack of transcribed data. The goal of this research is to propose a new data augmentation method to improve ASR models for agglutinative and low-resource languages. This novel data augmentation method generates both synthetic text and synthetic audio. Some experiments were conducted using the corpus of the Quechua language, which is an agglutinative and low-resource language. In this study, a sequence-to-sequence (seq2seq) model was applied to generate synthetic text, in addition to generating synthetic speech using a text-to-speech (TTS) model for Quechua. The results show that the new data augmentation method works well to improve the ASR model for Quechua. In this research, an 8.73% improvement in the word-error-rate (WER) of the ASR model is obtained using a combination of synthetic text and synthetic speech.

Motivation & Objective

• To address the critical challenge of limited transcribed speech data in training automatic speech recognition (ASR) models for low-resource languages.
• To improve ASR performance for agglutinative languages—specifically Quechua—by generating synthetic training data.
• To develop and evaluate a hybrid data augmentation framework combining sequence-to-sequence text generation and text-to-speech synthesis.
• To demonstrate the effectiveness of synthetic data in reducing word error rate (WER) in low-resource ASR scenarios.

Proposed method

• A sequence-to-sequence (seq2seq) model is trained to generate synthetic text from existing transcribed speech data in Quechua.
• A separate text-to-speech (TTS) model is trained to convert the generated synthetic text into synthetic audio.
• The synthetic text and audio pairs are combined with real training data to augment the ASR training set.
• The augmented dataset is used to fine-tune a pre-trained ASR model, improving its robustness and generalization.
• The method is evaluated using the Quechua language corpus, a low-resource, agglutinative language with limited transcribed data.
• The approach leverages both linguistic and acoustic data augmentation by generating semantically plausible, phonetically valid utterances.

Experimental results

Research questions

• RQ1 Can synthetic text and audio generation improve ASR performance in low-resource, agglutinative languages like Quechua?
• RQ2 How effective is a combined seq2seq + TTS data augmentation pipeline in reducing word error rate (WER) for ASR models?
• RQ3 Does the quality of synthetic data impact the generalization capability of ASR models in low-resource settings?
• RQ4 What is the relative contribution of synthetic text versus synthetic audio to ASR performance gains?

Key findings

• The proposed data augmentation method achieved an 8.73% relative improvement in word error rate (WER) on the Quechua ASR model.
• The combination of synthetic text and synthetic audio yielded the highest performance gain, outperforming using either modality alone.
• The seq2seq model successfully generated linguistically valid Quechua text, preserving morphological complexity typical of agglutinative languages.
• The TTS model produced natural-sounding synthetic speech that was effective for ASR fine-tuning.
• The method demonstrated robustness in low-resource settings where real transcribed data is scarce.
• The results confirm that synthetic data generation via TTS and seq2seq can effectively expand training data and improve ASR generalization.