[Paper Review] Hierarchical Diffusion Models for Singing Voice Neural Vocoder

Naoya Takahashi, Mayank Kumar+2 more|arXiv (Cornell University)|Oct 14, 2022

Music and Audio Processing4 citations

TL;DR

This paper proposes a hierarchical diffusion model for singing voice neural vocoders that uses multiple diffusion models at different sampling rates, conditioned on lower-rate waveforms and acoustic features. The method achieves state-of-the-art perceptual quality, outperforming PriorGrad and Parallel WaveGAN with similar computational costs, particularly excelling in pitch accuracy and naturalness for singing voices with vibrato.

ABSTRACT

Recent progress in deep generative models has improved the quality of neural vocoders in speech domain. However, generating a high-quality singing voice remains challenging due to a wider variety of musical expressions in pitch, loudness, and pronunciations. In this work, we propose a hierarchical diffusion model for singing voice neural vocoders. The proposed method consists of multiple diffusion models operating in different sampling rates; the model at the lowest sampling rate focuses on generating accurate low-frequency components such as pitch, and other models progressively generate the waveform at higher sampling rates on the basis of the data at the lower sampling rate and acoustic features. Experimental results show that the proposed method produces high-quality singing voices for multiple singers, outperforming state-of-the-art neural vocoders with a similar range of computational costs.

Motivation & Objective

• To address the challenge of generating high-quality singing voices due to complex musical expressions in pitch, loudness, and pronunciation.
• To overcome limitations of existing neural vocoders, which often produce unnatural pitch modulation or poor perceptual quality in singing voice synthesis.
• To develop a hierarchical diffusion framework that progressively generates waveforms from low to high sampling rates, enabling better modeling of singing voice characteristics.
• To improve sample quality and inference efficiency by conditioning higher-rate diffusion models on lower-rate generated waveforms and acoustic features.
• To validate the method's superiority in perceptual quality and objective metrics compared to state-of-the-art vocoders like PriorGrad and Parallel WaveGAN.

Proposed method

• The model employs multiple diffusion models trained independently at different sampling rates, from low to high, forming a hierarchical cascade.
• Each diffusion model at rate $f_s^i$ is conditioned on the acoustic features $c$ and the downsampled output $x_0^{i+1}$ from the next lower-rate model.
• During inference, the output of each lower-rate model is passed through an anti-aliasing filter $H^i$ before being used as conditioning for the higher-rate model.
• The forward process gradually adds noise to the original waveform, while the reverse process denoises it step-by-step using a learned denoising network $\mu_\theta(x_t, t)$.
• The method uses a data-dependent prior (as in PriorGrad) with frame-wise energy and spectral envelope to accelerate convergence and improve sample quality.
• The hierarchical structure enables the lowest-rate model to focus on pitch and fundamental frequency, while higher-rate models refine spectral and temporal details.

Experimental results

Research questions

• RQ1 Can a hierarchical diffusion model with multi-rate conditioning improve the perceptual quality of singing voice synthesis compared to single-rate diffusion models?
• RQ2 Does conditioning higher-rate diffusion models on lower-rate generated waveforms enhance pitch accuracy and naturalness in singing voices?
• RQ3 How does the hierarchical design compare to autoregressive or GAN-based vocoders in terms of quality and inference speed for singing voice generation?
• RQ4 To what extent does the use of a data-dependent prior improve convergence and sample quality in the hierarchical diffusion framework?
• RQ5 Can the hierarchical approach scale more efficiently than increasing model depth in baseline diffusion models like PriorGrad?

Key findings

• The proposed HPG-2 model achieves a mean opinion score (MOS) of 3.95 ± 0.13, significantly outperforming PriorGrad (3.60 ± 0.12) and Parallel WaveGAN (2.15 ± 0.13).
• In preference testing, 85.3% of raters preferred HPG-2 over PriorGrad, indicating strong perceptual superiority.
• HPG-2 achieves a pitch mean absolute error (PMAE) of 1.82, lower than PriorGrad (1.80) and significantly better than PWG (3.12), indicating improved pitch tracking.
• The model reduces voicing decision error (VDE) to 3.47, outperforming PWG (5.61) and showing better phonetic clarity.
• The 3-stage HPG-3 model further improves metrics (PMAE: 1.67, VDE: 3.32, MR-STFT: 1.07, MCD: 8.12) with only a 30% increase in computational cost.
• Ablation studies confirm that the lower-rate waveform $x_0^2$ is essential for low-frequency component generation, while the mel-spectrogram $c$ primarily conditions high-frequency details.