[Paper Review] PortaSpeech: Portable and High-Quality Generative Text-to-Speech

Yi Ren, Jinglin Liu+1 more|arXiv (Cornell University)|Sep 30, 2021

Speech Recognition and Synthesis31 references6 citations

TL;DR

PortaSpeech is a lightweight, high-quality non-autoregressive text-to-speech model that combines a variational autoencoder with an enhanced prior and a flow-based post-net to generate expressive, natural-sounding speech with rich prosody and fine-grained spectral details. It achieves state-of-the-art subjective and objective performance with only 6.7M parameters—4× smaller and 3× more memory-efficient than FastSpeech 2—while maintaining high quality through a novel linguistic encoder with mixture alignment and grouped parameter sharing in the post-net.

ABSTRACT

Non-autoregressive text-to-speech (NAR-TTS) models such as FastSpeech 2 and Glow-TTS can synthesize high-quality speech from the given text in parallel. After analyzing two kinds of generative NAR-TTS models (VAE and normalizing flow), we find that: VAE is good at capturing the long-range semantics features (e.g., prosody) even with small model size but suffers from blurry and unnatural results; and normalizing flow is good at reconstructing the frequency bin-wise details but performs poorly when the number of model parameters is limited. Inspired by these observations, to generate diverse speech with natural details and rich prosody using a lightweight architecture, we propose PortaSpeech, a portable and high-quality generative text-to-speech model. Specifically, 1) to model both the prosody and mel-spectrogram details accurately, we adopt a lightweight VAE with an enhanced prior followed by a flow-based post-net with strong conditional inputs as the main architecture. 2) To further compress the model size and memory footprint, we introduce the grouped parameter sharing mechanism to the affine coupling layers in the post-net. 3) To improve the expressiveness of synthesized speech and reduce the dependency on accurate fine-grained alignment between text and speech, we propose a linguistic encoder with mixture alignment combining hard inter-word alignment and soft intra-word alignment, which explicitly extracts word-level semantic information. Experimental results show that PortaSpeech outperforms other TTS models in both voice quality and prosody modeling in terms of subjective and objective evaluation metrics, and shows only a slight performance degradation when reducing the model parameters to 6.7M (about 4x model size and 3x runtime memory compression ratio compared with FastSpeech 2). Our extensive ablation studies demonstrate that each design in PortaSpeech is effective.

Motivation & Objective

• Address the trade-off between model efficiency and speech quality in non-autoregressive text-to-speech (NAR-TTS) systems.
• Enable high-fidelity, expressive speech synthesis with minimal computational and memory overhead for edge deployment.
• Reduce dependency on precise phoneme-level alignments by introducing a hybrid alignment mechanism in the linguistic encoder.
• Improve prosody modeling and spectral detail reconstruction using a hybrid architecture combining VAE and normalizing flow.
• Develop a compact, efficient model that maintains high performance even under extreme parameter and memory constraints.

Proposed method

• Adopt a two-stage generative architecture: a lightweight variational autoencoder (VAE) with an enhanced prior for modeling long-range prosodic structures, followed by a flow-based post-net for fine-grained mel-spectrogram detail refinement.
• Introduce a grouped parameter sharing mechanism in the affine coupling layers of the flow-based post-net to reduce model size and memory footprint without significant performance loss.
• Design a linguistic encoder with mixture alignment that combines hard word-level alignment and soft phoneme-level alignment to improve duration prediction and reduce reliance on precise phoneme-level alignments.
• Use a VAE with a non-Gaussian, enhanced prior distribution to better capture complex prosodic patterns, especially in small models.
• Train the end-to-end model using a combination of reconstruction loss, KL divergence, and flow-based likelihood maximization to optimize both quality and diversity.
• Apply knowledge distillation or other regularization techniques implicitly through the enhanced prior and flow-based refinement to stabilize training and improve generalization.

Experimental results

Research questions

• RQ1 How can a non-autoregressive TTS model achieve high-quality speech synthesis with minimal model size and memory footprint?
• RQ2 What are the respective strengths and weaknesses of VAE-based and normalizing flow-based architectures in modeling prosody and spectral details in TTS?
• RQ3 Can a hybrid architecture combining VAE and normalizing flow outperform standalone models in both voice quality and prosody modeling?
• RQ4 Does a mixture alignment mechanism in the linguistic encoder improve prosody modeling and reduce dependency on fine-grained alignment supervision?
• RQ5 Can significant model compression be achieved without substantial degradation in speech quality, especially in terms of naturalness and expressiveness?

Key findings

• PortaSpeech achieves state-of-the-art performance in both subjective (CMOS) and objective evaluation metrics, outperforming FastSpeech 2 and Glow-TTS in voice quality and prosody modeling.
• With only 6.7M parameters, PortaSpeech reduces model size by about 4× and memory footprint by about 3× compared to FastSpeech 2, with only a slight performance drop.
• The enhanced prior in the VAE significantly improves prosody modeling, as shown by a 0.194-point increase in CMOS-P when compared to a simple Gaussian prior.
• The flow-based post-net improves audio quality by 0.458 points in CMOS-Q compared to removing the post-net, and outperforms a standard convolutional post-net.
• Mixture alignment in the linguistic encoder reduces average absolute duration error by 40.4ms at the word level and 0.44s at the sentence level compared to phoneme-level hard alignment.
• Ablation studies confirm that each component—enhanced prior, flow-based post-net, and mixture alignment—contributes significantly to the overall performance, with the full model achieving the highest CMOS scores.