[Paper Review] Almost Unsupervised Text to Speech and Automatic Speech Recognition

Yi Ren, Xu Tan+4 more|arXiv (Cornell University)|May 13, 2019

Speech Recognition and Synthesis40 citations

TL;DR

The paper presents an almost unsupervised framework that jointly trains TTS and ASR using a few hundred paired samples plus large amounts of unpaired data, via denoising auto-encoders, dual transformation, bidirectional sequence modeling, all in a unified Transformer-based model.

ABSTRACT

Text to speech (TTS) and automatic speech recognition (ASR) are two dual tasks in speech processing and both achieve impressive performance thanks to the recent advance in deep learning and large amount of aligned speech and text data. However, the lack of aligned data poses a major practical problem for TTS and ASR on low-resource languages. In this paper, by leveraging the dual nature of the two tasks, we propose an almost unsupervised learning method that only leverages few hundreds of paired data and extra unpaired data for TTS and ASR. Our method consists of the following components: (1) a denoising auto-encoder, which reconstructs speech and text sequences respectively to develop the capability of language modeling both in speech and text domain; (2) dual transformation, where the TTS model transforms the text $y$ into speech $\hat{x}$, and the ASR model leverages the transformed pair $(\hat{x},y)$ for training, and vice versa, to boost the accuracy of the two tasks; (3) bidirectional sequence modeling, which addresses error propagation especially in the long speech and text sequence when training with few paired data; (4) a unified model structure, which combines all the above components for TTS and ASR based on Transformer model. Our method achieves 99.84% in terms of word level intelligible rate and 2.68 MOS for TTS, and 11.7% PER for ASR on LJSpeech dataset, by leveraging only 200 paired speech and text data (about 20 minutes audio), together with extra unpaired speech and text data.

Motivation & Objective

• Address the data scarcity problem in TTS and ASR for low-resource languages.
• Leverage the dual nature of TTS and ASR to improve both tasks with minimal paired data.
• Develop a unified Transformer-based architecture that can handle speech and text in both directions.
• Mitigate error propagation in long sequences via bidirectional sequence modeling.

Proposed method

• Use denoising auto-encoders to learn language modeling in both speech and text domains from unpaired data.
• Introduce dual transformation (back-translation inspired) to train TTS from ASR outputs and ASR from TTS outputs.
• Employ bidirectional sequence modeling to reduce error propagation by generating left-to-right and right-to-left sequences.
• Adopt a unified Transformer-based model with shared components for TTS and ASR, including speech and text input/output modules.
• Train with a combination of denoising auto-encoder losses, dual transformation losses, and supervised losses on the limited paired data.

Experimental results

Research questions

• RQ1 Can TTS and ASR be effectively trained in an almost unsupervised setting using a few hundred paired samples and abundant unpaired data?
• RQ2 Does dual transformation leveraging the dual nature of TTS and ASR improve both tasks compared to using only paired data?
• RQ3 Can bidirectional sequence modeling mitigate error propagation in long speech/text sequences under low-resource conditions?
• RQ4 Is a unified Transformer-based architecture capable of handling both TTS and ASR inputs/outputs with shared parameters?

Key findings

• Achieves 99.84% word-level intelligibility on TTS using only ~200 paired samples plus unpaired data (vs near 0 intelligibility with 200 paired data alone).
• Attains 2.68 MOS for TTS and 11.7% PER for ASR on LJSpeech under the same data conditions, outperforming the 200-paired-data baseline.
• Bidirectional sequence modeling improves MOS by 0.40 and reduces PER by 3.6 percentage points over the baseline with DT alone.
• Ablation shows DAE boosts ASR PER from 72.3% to 52.0%; adding DT lowers PER to 15.3% (and MOS to 2.11); adding bidirectional modeling yields further gains to 11.7% PER and 2.51 MOS.
• With more paired data (e.g., 500 pairs), PER improves to 4.4%, approaching the supervised lower bound (2.5%).
• Visualization of mel-spectrograms indicates finer detail recovery with bidirectional modeling, especially at sequence ends.