[Paper Review] Exploring Machine Speech Chain for Domain Adaptation and Few-Shot Speaker Adaptation

Fengpeng Yue, Yan Deng+2 more|arXiv (Cornell University)|Apr 8, 2021

Speech Recognition and Synthesis35 references4 citations

TL;DR

This paper proposes a machine speech chain framework that leverages TTS→ASR joint training for domain and few-shot speaker adaptation using only text data from the target domain. By generating synthetic speech from target-domain text and using ASR feedback, it achieves a 10% relative WER reduction in E2E ASR and 51.5% relative WER reduction in TTS-generated speech on TED-LIUM, with additional 2.6% relative WER gain via few-shot speaker adaptation.

ABSTRACT

Machine Speech Chain, which integrates both end-to-end (E2E) automatic speech recognition (ASR) and text-to-speech (TTS) into one circle for joint training, has been proven to be effective in data augmentation by leveraging large amounts of unpaired data. In this paper, we explore the TTS->ASR pipeline in speech chain to do domain adaptation for both neural TTS and E2E ASR models, with only text data from target domain. We conduct experiments by adapting from audiobook domain (LibriSpeech) to presentation domain (TED-LIUM), there is a relative word error rate (WER) reduction of 10% for the E2E ASR model on the TED-LIUM test set, and a relative WER reduction of 51.5% in synthetic speech generated by neural TTS in the presentation domain. Further, we apply few-shot speaker adaptation for the E2E ASR by using a few utterances from target speakers in an unsupervised way, results in additional gains.

Motivation & Objective

• To address text-domain and speaker variability mismatches in E2E ASR and neural TTS using a joint learning framework.
• To enable domain adaptation for ASR and TTS without requiring paired speech-text data from the target domain.
• To explore unsupervised few-shot speaker adaptation by leveraging synthetic speech generation from TTS and ASR feedback.
• To evaluate the effectiveness of the machine speech chain in improving robustness and performance on out-of-domain text and unseen speakers.

Proposed method

• The method employs a TTS→ASR pipeline within a machine speech chain, where TTS generates synthetic speech from target-domain text, and ASR provides feedback via cross-entropy loss to improve both models.
• Joint training updates both ASR and TTS models using the ASR loss on synthetic speech, enabling the models to learn target-domain acoustic and linguistic representations.
• For domain adaptation, the TTS model is fine-tuned using text from the target domain (TED-LIUM), and the ASR model is trained on the resulting synthetic speech, reducing domain mismatch.
• For few-shot speaker adaptation, a few reference utterances from target speakers are used to condition the TTS model, which then generates speaker-specific synthetic speech for ASR fine-tuning.
• The framework uses attention-based E2E ASR and Tacotron2-based TTS, with guided attention and cross-entropy loss for training stability and performance.
• Training strategies include joint optimization of both models and selective fine-tuning of only ASR while keeping TTS fixed, to evaluate impact on performance and robustness.

Experimental results

Research questions

• RQ1 Can the TTS→ASR pipeline in a machine speech chain effectively reduce domain mismatch in ASR and TTS using only target-domain text?
• RQ2 How does joint training of TTS and ASR improve robustness and reduce attention errors in neural TTS for out-of-domain text?
• RQ3 To what extent can unsupervised few-shot speaker adaptation be achieved via synthetic speech generation and ASR feedback in the speech chain framework?
• RQ4 Does joint training with synthetic data from the target domain improve ASR performance on real target-domain test sets without degrading source-domain performance?

Key findings

• The proposed domain adaptation method achieved a 10% relative WER reduction in E2E ASR on the TED-LIUM test set, from 25.9% to 23.3%.
• Neural TTS showed a 51.5% relative WER reduction in synthetic speech generated from target-domain text, indicating significant improvement in attention robustness.
• The performance of TTS on the source domain (LibriSpeech) also improved by 16.9% relative WER, showing generalization benefits from joint training.
• Using only one reference utterance for speaker adaptation reduced ASR WER by 2.6% relative, with further gains when five references were used.
• Joint training of both ASR and TTS yielded better TTS robustness, while fine-tuning only ASR gave slightly better ASR performance, suggesting a trade-off between model improvements.
• Visual analysis confirmed improved attention alignment in Tacotron2 after joint training, especially for out-of-domain text, validating reduced attention errors.