[Paper Review] LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search

Renqian Luo, Xu Tan+6 more|arXiv (Cornell University)|Feb 8, 2021

Speech Recognition and Synthesis24 references4 citations

TL;DR

LightSpeech proposes a neural architecture search (NAS)-driven approach to automatically design lightweight and fast text-to-speech models based on FastSpeech 2. By profiling model components and designing a targeted search space with depthwise separable convolutions and efficient blocks, it achieves 15x model compression, 16x fewer MACs, and 6.5x faster inference on CPU with on-par voice quality.

ABSTRACT

Text to speech (TTS) has been broadly used to synthesize natural and intelligible speech in different scenarios. Deploying TTS in various end devices such as mobile phones or embedded devices requires extremely small memory usage and inference latency. While non-autoregressive TTS models such as FastSpeech have achieved significantly faster inference speed than autoregressive models, their model size and inference latency are still large for the deployment in resource constrained devices. In this paper, we propose LightSpeech, which leverages neural architecture search~(NAS) to automatically design more lightweight and efficient models based on FastSpeech. We first profile the components of current FastSpeech model and carefully design a novel search space containing various lightweight and potentially effective architectures. Then NAS is utilized to automatically discover well performing architectures within the search space. Experiments show that the model discovered by our method achieves 15x model compression ratio and 6.5x inference speedup on CPU with on par voice quality. Audio demos are provided at https://speechresearch.github.io/lightspeech.

Motivation & Objective

• To address the challenge of deploying high-quality TTS models on resource-constrained devices like mobile phones and embedded systems.
• To reduce model size, inference latency, and computational cost (MACs) without sacrificing voice quality.
• To overcome the limitations of manual model compression, which often leads to significant performance degradation.
• To develop a search space and NAS pipeline tailored for sequence-to-sequence TTS models, particularly non-autoregressive architectures like FastSpeech 2.
• To demonstrate that NAS can outperform human-designed lightweight models in both efficiency and performance on TTS tasks.

Proposed method

• Conducted a detailed profiling of FastSpeech 2 components to identify memory and latency bottlenecks, especially in the encoder, decoder, and variance predictors.
• Designed a novel search space that includes depthwise separable convolutions (SepConv), variable kernel sizes, and reduced block counts, while preserving model capacity.
• Selected GBDT-NAS as the search algorithm due to its efficiency and compatibility with the chain-structured search space.
• Used accuracy prediction via gradient-boosted decision trees to efficiently evaluate candidate architectures without full training.
• Optimized the search space to include only lightweight, effective components such as SepConv with kernel sizes 5, 9, 13, 17, 21, 25.
• Trained and evaluated the final architecture on standard TTS metrics including mel-spectrogram loss and CMOS for voice quality.

Experimental results

Research questions

• RQ1 Can neural architecture search effectively discover lightweight TTS architectures that outperform manually designed models in both efficiency and quality?
• RQ2 How does the choice of search space design impact the performance and efficiency of NAS in TTS models?
• RQ3 To what extent can depthwise separable convolutions and architectural pruning reduce model size and MACs without degrading voice quality?
• RQ4 Can NAS-based compression achieve significant speedup on CPU while maintaining on-par audio quality compared to standard FastSpeech 2?
• RQ5 How does the performance of NAS-discovered models compare to hand-crafted lightweight variants of FastSpeech 2 in terms of inference latency and model size?

Key findings

• The NAS-discovered LightSpeech model achieves a 15x model compression ratio (1.8M parameters vs. 27M in FastSpeech 2).
• LightSpeech reduces computation cost by 16x, with only 0.76G MACs compared to FastSpeech 2’s 12.50G MACs.
• The model achieves a 6.5x speedup in inference time on CPU, reducing RTF from 6.1×10⁻² to 9.3×10⁻³.
• LightSpeech maintains on-par voice quality, with a CMOS score of 0.2561 compared to FastSpeech 2’s 0.2575.
• Random sampling from the search space already outperforms a manually designed lightweight FastSpeech 2* (0.2753 vs. 0.2956 loss), proving the quality of the search space.
• The final architecture uses a sequence of SepConv layers with kernel sizes 5, 25, 13, 9 in the encoder and 17, 21, 9, 13 in the decoder, all with hidden size 256.