[Paper Review] UnivNet: A Neural Vocoder with Multi-Resolution Spectrogram Discriminators for High-Fidelity Waveform Generation

Won Jang, Dan Lim+3 more|arXiv (Cornell University)|Jun 15, 2021

Speech and Audio Processing29 references4 citations

TL;DR

UnivNet is a real-time neural vocoder that uses full-band mel-spectrograms and a multi-resolution spectrogram discriminator (MRSD) to generate high-fidelity waveforms, effectively mitigating the over-smoothing problem common in full-band models. It achieves state-of-the-art subjective and objective performance across seen and unseen speakers, with a lightweight version generating waveforms 227x faster than real time.

ABSTRACT

Most neural vocoders employ band-limited mel-spectrograms to generate waveforms. If full-band spectral features are used as the input, the vocoder can be provided with as much acoustic information as possible. However, in some models employing full-band mel-spectrograms, an over-smoothing problem occurs as part of which non-sharp spectrograms are generated. To address this problem, we propose UnivNet, a neural vocoder that synthesizes high-fidelity waveforms in real time. Inspired by works in the field of voice activity detection, we added a multi-resolution spectrogram discriminator that employs multiple linear spectrogram magnitudes computed using various parameter sets. Using full-band mel-spectrograms as input, we expect to generate high-resolution signals by adding a discriminator that employs spectrograms of multiple resolutions as the input. In an evaluation on a dataset containing information on hundreds of speakers, UnivNet obtained the best objective and subjective results among competing models for both seen and unseen speakers. These results, including the best subjective score for text-to-speech, demonstrate the potential for fast adaptation to new speakers without a need for training from scratch.

Motivation & Objective

• To address the over-smoothing problem in neural vocoders that use full-band mel-spectrograms, which leads to loss of high-frequency detail and audible artifacts.
• To improve spectral resolution and waveform fidelity by introducing a multi-resolution spectrogram discriminator (MRSD) that analyzes multiple spectrogram scales.
• To enable fast adaptation to new speakers without retraining by leveraging a universal vocoder design.
• To achieve high-fidelity, real-time inference with minimal computational cost and strong generalization across diverse speakers.

Proposed method

• The generator uses a non-autoregressive architecture inspired by MelGAN, incorporating location-variable convolution (LVC) layers for efficient local feature modeling.
• The generator is conditioned on full-band mel-spectrograms and produces waveforms through transposed convolutions with residual blocks.
• A multi-resolution spectrogram discriminator (MRSD) processes multiple linear spectrogram magnitudes computed with varying STFT parameters to enhance spectral resolution.
• The model combines a multi-period waveform discriminator (MPWD) and the MRSD to model both temporal periodicity and spectral detail.
• The generator is trained with adversarial loss and a multi-scale STFT loss, with the MRSD providing high-frequency resolution feedback.
• The architecture is optimized for inference speed, with a lightweight variant (UnivNet-c16) achieving 227x real-time generation speed.

Experimental results

Research questions

• RQ1 Can a neural vocoder using full-band mel-spectrograms achieve high-fidelity waveform generation without suffering from over-smoothing?
• RQ2 Does a multi-resolution spectrogram discriminator improve spectral resolution and reduce artifacts in generated waveforms?
• RQ3 Can a single vocoder model generalize effectively to unseen speakers without fine-tuning?
• RQ4 How does the combination of multi-period and multi-resolution discriminators enhance waveform quality and temporal-spectral fidelity?
• RQ5 What is the trade-off between model complexity, inference speed, and subjective quality in full-band neural vocoders?

Key findings

• UnivNet-c16 achieved a mean opinion score (MOS) of 3.92 for seen speakers and 3.79 for unseen speakers, outperforming MelGAN, Parallel WaveGAN, and HiFi-GAN.
• In text-to-speech evaluation, UnivNet-c32 achieved the highest MOS of 4.07, demonstrating strong zero-shot generalization to new speakers.
• The ablation study confirmed that removing the MRSD caused significant over-smoothing and audible metallic artifacts, reducing MOS by over 0.5 points.
• UnivNet-c16 generated waveforms at 227.27× real-time speed, significantly faster than HiFi-GAN (135.14×) and MelGAN (294.11×), while maintaining high quality.
• UnivNet-c32 achieved the best objective scores with PESQ of 3.70 and RMSE of 0.294 for unseen speakers, outperforming all baselines.
• The combination of MPWD and MRSD (MSWD) yielded the highest MOS (3.90), confirming the complementary benefits of multi-scale temporal and spectral discrimination.