[Paper Review] WaveTTS: Tacotron-based TTS with Joint Time-Frequency Domain Loss

Rui Liu, Berrak Şişman+3 more|arXiv (Cornell University)|Feb 2, 2020

Speech Recognition and Synthesis44 references4 citations

TL;DR

WaveTTS proposes a novel Tacotron-based text-to-speech framework that jointly optimizes for both time-domain waveform quality and frequency-domain acoustic features using a dual loss function: a time-domain loss (SI-SDR) and a standard Mel-spectrogram loss. This joint training significantly improves speech quality, outperforming baselines in both MOS and preference tests, and is the first implementation of such a scheme in Tacotron-based TTS.

ABSTRACT

Tacotron-based text-to-speech (TTS) systems directly synthesize speech from text input. Such frameworks typically consist of a feature prediction network that maps character sequences to frequency-domain acoustic features, followed by a waveform reconstruction algorithm or a neural vocoder that generates the time-domain waveform from acoustic features. As the loss function is usually calculated only for frequency-domain acoustic features, that doesn't directly control the quality of the generated time-domain waveform. To address this problem, we propose a new training scheme for Tacotron-based TTS, referred to as WaveTTS, that has 2 loss functions: 1) time-domain loss, denoted as the waveform loss, that measures the distortion between the natural and generated waveform; and 2) frequency-domain loss, that measures the Mel-scale acoustic feature loss between the natural and generated acoustic features. WaveTTS ensures both the quality of the acoustic features and the resulting speech waveform. To our best knowledge, this is the first implementation of Tacotron with joint time-frequency domain loss. Experimental results show that the proposed framework outperforms the baselines and achieves high-quality synthesized speech.

Motivation & Objective

• Address the mismatch between frequency-domain loss optimization and actual time-domain waveform quality in Tacotron-based TTS.
• Improve speech synthesis quality by directly supervising the time-domain waveform during training.
• Investigate the effectiveness of combining time-domain and frequency-domain losses in a single training objective.
• Demonstrate that joint optimization leads to better performance even when using non-neural vocoders like Griffin-Lim at inference.
• Establish a new training paradigm for Tacotron that enhances generalization and robustness across different vocoders.

Proposed method

• Introduce a joint training scheme that applies both frequency-domain loss (Mel-spectrogram loss) and time-domain loss (SI-SDR) during model optimization.
• Use Griffin-Lim with iterative phase reconstruction to generate time-domain waveforms from predicted Mel-spectrograms for time-domain loss computation.
• Compute the time-domain loss using scale-invariant signal-to-distortion ratio (SI-SDR), which is robust to amplitude and phase mismatches.
• Train the feature prediction network end-to-end with both losses, enabling it to learn features that yield high-quality waveforms.
• Apply the same joint loss during training regardless of the inference vocoder (Griffin-Lim or WaveNet), ensuring consistent optimization.
• Use a standard Tacotron architecture with encoder, attention-based decoder, and post-net, but modify the loss function to include waveform-level supervision.

Experimental results

Research questions

• RQ1 Can joint optimization over time and frequency domains improve the perceptual quality of synthesized speech in Tacotron-based TTS?
• RQ2 Does incorporating a time-domain loss function during training reduce waveform artifacts and improve naturalness, even when using deterministic vocoders like Griffin-Lim?
• RQ3 How does the proposed joint loss perform when the inference vocoder is changed from Griffin-Lim to a neural vocoder like WaveNet?
• RQ4 What is the impact of the number of Griffin-Lim iterations on the time-domain loss and final speech quality?
• RQ5 Can a model trained with joint loss generalize better across different vocoders without retraining?

Key findings

• WaveTTS-GL achieved a mean opinion score (MOS) of 3.30, significantly outperforming Tacotron-GL, which scored 3.18.
• WaveTTS-WN, when used with a WaveNet vocoder at inference, outperformed Tacotron-WN, demonstrating the benefit of joint training even with neural vocoders.
• Preference tests showed that WaveTTS-GL was preferred over Tacotron-GL in 72% of A/B comparisons, with 95% confidence interval.
• WaveTTS-WN achieved higher preference scores than Tacotron-WN, confirming the generalization benefit of joint training.
• The A/B test revealed that 1 iteration of Griffin-Lim yielded better speech quality than 2 iterations, suggesting optimal trade-off in waveform reconstruction during training.
• The proposed joint loss framework is effective regardless of the inference vocoder, indicating robustness and generalization across different waveform generation methods.