[Paper Review] Non-Attentive Tacotron: Robust and Controllable Neural TTS Synthesis Including Unsupervised Duration Modeling

Jonathan Shen, Jia Ye+5 more|arXiv (Cornell University)|Oct 8, 2020

Neural Networks and ApplicationsComputer Science52 references73 citations

TL;DR

This paper replaces attention in Tacotron 2 with an explicit duration predictor and Gaussian upsampling, enabling robust, controllable TTS with supervised, semi-supervised, or unsupervised duration modeling.

ABSTRACT

This paper presents Non-Attentive Tacotron based on the Tacotron 2 text-to-speech model, replacing the attention mechanism with an explicit duration predictor. This improves robustness significantly as measured by unaligned duration ratio and word deletion rate, two metrics introduced in this paper for large-scale robustness evaluation using a pre-trained speech recognition model. With the use of Gaussian upsampling, Non-Attentive Tacotron achieves a 5-scale mean opinion score for naturalness of 4.41, slightly outperforming Tacotron 2. The duration predictor enables both utterance-wide and per-phoneme control of duration at inference time. When accurate target durations are scarce or unavailable in the training data, we propose a method using a fine-grained variational auto-encoder to train the duration predictor in a semi-supervised or unsupervised manner, with results almost as good as supervised training.

Motivation & Objective

• Motivate robustness issues in attention-based neural TTS and reduce risk of failures like repetition or long pauses.
• Introduce Non-Attentive Tacotron (NAT) that uses a duration predictor and Gaussian upsampling instead of attention.
• Enable training with supervised, semi-supervised, or unsupervised duration information via FVAE-based alignment.
• Provide methods to control utterance-wide pace and per-phoneme duration at inference while maintaining quality.
• Propose robust automated evaluation metrics (UDR and WDR) for large-scale robustness assessment.

Proposed method

• Replace Tacotron 2 attention with a duration predictor and Gaussian upsampling to upsample encoder outputs.
• Predict per-token duration d and a range parameter sigma for Gaussian upsampling.
• Upsample encoder outputs via Gaussian mixtures centered at token segments to form aligned inputs for the decoder.
• Train with a loss combining mel-spectrogram reconstruction and duration prediction (L_spec and L_dur).
• Support semi-supervised/unsupervised duration modeling using a FVAE that extracts token-aligned latent features from the target spectrogram to inform duration prediction.
• Allow inference-time control of utterance-wide pace and per-phoneme timing by manipulating predicted durations.

Experimental results

Research questions

• RQ1 Can explicit duration modeling via a duration predictor and Gaussian upsampling improve TTS robustness over attention-based Tacotron 2?
• RQ2 How does unsupervised or semi-supervised duration modeling compare to fully supervised training in terms of naturalness and robustness?
• RQ3 To what extent can NAT provide controllable pacing at utterance-wide and per-phoneme levels without sacrificing quality?
• RQ4 What are effective metrics for large-scale robustness in TTS beyond MOS (e.g., UDR and WDR)?

Key findings

• NAT with Gaussian upsampling achieves naturalness on par with Tacotron 2 (GMMA) in MOS tests.
• Gaussian upsampling significantly improves robustness over vanilla upsampling and attention-based baselines.
• Supervised NAT yields very high robustness (low UDR/WDR) and close-to-ground-truth MOS.
• Semi-supervised and unsupervised duration modeling with a FVAE approach retain much of the naturalness and robustness, outperforming plain unsupervised methods without FVAE.
• Autoregressive decoder remains critical for high-quality synthesis; non-autoregressive decoders underperform NAT’s naturalness.
• NAT enables both utterance-wide and fine-grained pace control at inference without losing quality in supervised settings.