[Paper Review] Multi-band MelGAN: Faster Waveform Generation for High-Quality Text-to-Speech

Geng Yang, Shan Yang+4 more|arXiv (Cornell University)|May 11, 2020

Speech Recognition and SynthesisComputer Science26 references21 citations

TL;DR

This paper proposes Multi-band MelGAN (MB-MelGAN), a faster and higher-quality neural vocoder for text-to-speech by expanding the receptive field, replacing feature matching loss with multi-resolution STFT loss, and introducing multi-band waveform generation. The model achieves a MOS of 4.34 in waveform generation and 4.22 in TTS, with only 1.91M parameters and 0.95 GFLOPS, enabling a real-time factor of 0.03 on CPU—7x faster than original MelGAN while maintaining high audio quality.

ABSTRACT

In this paper, we propose multi-band MelGAN, a much faster waveform generation model targeting to high-quality text-to-speech. Specifically, we improve the original MelGAN by the following aspects. First, we increase the receptive field of the generator, which is proven to be beneficial to speech generation. Second, we substitute the feature matching loss with the multi-resolution STFT loss to better measure the difference between fake and real speech. Together with pre-training, this improvement leads to both better quality and better training stability. More importantly, we extend MelGAN with multi-band processing: the generator takes mel-spectrograms as input and produces sub-band signals which are subsequently summed back to full-band signals as discriminator input. The proposed multi-band MelGAN has achieved high MOS of 4.34 and 4.22 in waveform generation and TTS, respectively. With only 1.91M parameters, our model effectively reduces the total computational complexity of the original MelGAN from 5.85 to 0.95 GFLOPS. Our Pytorch implementation, which will be open-resourced shortly, can achieve a real-time factor of 0.03 on CPU without hardware specific optimization.

Motivation & Objective

• To improve the quality and training stability of MelGAN for high-fidelity speech synthesis.
• To reduce computational complexity and accelerate inference speed without sacrificing audio quality.
• To address audible artifacts such as jitter and metallic sounds in original MelGAN.
• To enable efficient, real-time inference on CPU for practical TTS deployment.
• To validate the effectiveness of multi-band processing in reducing model complexity while preserving perceptual quality.

Proposed method

• Expanding the receptive field of the MelGAN generator to improve long-range speech dependency modeling.
• Replacing the original feature matching loss with multi-resolution STFT loss to better capture spectral and temporal differences between real and generated speech.
• Introducing a multi-band processing framework where the generator predicts sub-band waveforms that are summed to form full-band outputs.
• Applying multi-resolution STFT loss both on sub-band and full-band waveforms to enhance perceptual quality and training stability.
• Using pre-training to improve convergence and audio quality during adversarial training.
• Implementing a single shared generator network across all sub-bands to reduce parameter count and computational cost.

Experimental results

Research questions

• RQ1 Can expanding the receptive field of MelGAN significantly improve speech quality and training stability?
• RQ2 Does replacing feature matching loss with multi-resolution STFT loss lead to better perceptual quality and faster convergence?
• RQ3 Can multi-band processing reduce computational complexity while maintaining high-fidelity speech generation?
• RQ4 How does combining sub-band and full-band multi-resolution STFT losses affect audio quality and training dynamics?
• RQ5 Can MB-MelGAN achieve high MOS with minimal parameters and low real-time factor on CPU for real-time TTS applications?

Key findings

• MB-MelGAN achieves a mean opinion score (MOS) of 4.34 in waveform generation, a significant improvement over the original MelGAN’s 3.98.
• In text-to-speech tasks, MB-MelGAN achieves a MOS of 4.22, outperforming the original MelGAN (3.87) and FB-MelGAN (4.18).
• The model reduces computational complexity from 5.85 GFLOPS (original MelGAN) to 0.95 GFLOPS, enabling a real-time factor of 0.03 on CPU.
• The multi-band design reduces training time by approximately 50% compared to full-band models while maintaining or improving audio quality.
• Combining sub-band and full-band multi-resolution STFT losses improves both audio quality and training stability, leading to faster convergence.
• The model achieves high-quality speech synthesis with only 1.91 million parameters, making it highly efficient for deployment on resource-constrained devices.