[Paper Review] DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism

Jinglin Liu, Chengxi Li+3 more|arXiv (Cornell University)|May 6, 2021

Music and Audio Processing35 citations

TL;DR

DiffSinger uses a diffusion probabilistic model for SVS to reduce over-smoothing and stabilize training, and introduces a shallow diffusion mechanism to start the reverse process from an intersection point with a simple mel-spectrogram decoder, improving quality and inference speed.

ABSTRACT

Singing voice synthesis (SVS) systems are built to synthesize high-quality and expressive singing voice, in which the acoustic model generates the acoustic features (e.g., mel-spectrogram) given a music score. Previous singing acoustic models adopt a simple loss (e.g., L1 and L2) or generative adversarial network (GAN) to reconstruct the acoustic features, while they suffer from over-smoothing and unstable training issues respectively, which hinder the naturalness of synthesized singing. In this work, we propose DiffSinger, an acoustic model for SVS based on the diffusion probabilistic model. DiffSinger is a parameterized Markov chain that iteratively converts the noise into mel-spectrogram conditioned on the music score. By implicitly optimizing variational bound, DiffSinger can be stably trained and generate realistic outputs. To further improve the voice quality and speed up inference, we introduce a shallow diffusion mechanism to make better use of the prior knowledge learned by the simple loss. Specifically, DiffSinger starts generation at a shallow step smaller than the total number of diffusion steps, according to the intersection of the diffusion trajectories of the ground-truth mel-spectrogram and the one predicted by a simple mel-spectrogram decoder. Besides, we propose boundary prediction methods to locate the intersection and determine the shallow step adaptively. The evaluations conducted on a Chinese singing dataset demonstrate that DiffSinger outperforms state-of-the-art SVS work. Extensional experiments also prove the generalization of our methods on text-to-speech task (DiffSpeech). Audio samples: https://diffsinger.github.io. Codes: https://github.com/MoonInTheRiver/DiffSinger. The old title of this work: "Diffsinger: Diffusion acoustic model for singing voice synthesis".

Motivation & Objective

• Address over-smoothing and unstable training in SVS acoustic models.
• Propose a diffusion-based SVS model (DiffSinger) trained via ELBO without adversarial loss.
• Introduce a shallow diffusion mechanism to leverage prior simple-loss knowledge.
• Develop a boundary prediction network to adaptively determine the shallow diffusion start step.
• Demonstrate generalization of the approach through DiffSpeech for TTS.

Proposed method

• Model SVS as a conditional diffusion process producing mel-spectrograms from music scores.
• Train the diffusion model by minimizing a variational lower bound (ELBO) with a denoiser predicting noise.
• In inference, start from a shallow diffusion step k using an auxiliary mel-decoder to generate M_k and then perform k denoising steps.
• Introduce a boundary predictor to adaptively determine the intersection step k between diffusion trajectories of ground-truth M and auxiliary \widetilde{M}\u007f.
• Adopt an encoder with lyrics, length regulator, and pitch components to condition the diffusion process on the music score.
• Use a non-causal WaveNet-based denoiser and an auxiliary FFT-based mel-spectrogram decoder for M generation.
• Provide an extensional DiffSpeech adaptation for English TTS tasks.

Experimental results

Research questions

• RQ1 Can diffusion models produce natural-sounding singing voice from a music score without adversarial training?
• RQ2 Does a shallow diffusion start point improve synthesis quality and inference speed in SVS?
• RQ3 Can a boundary predictor adaptively determine the optimal shallow diffusion step k?
• RQ4 Does the approach generalize to TTS (DiffSpeech) beyond SVS?

Key findings

• DiffSinger outperforms state-of-the-art SVS systems on a Chinese singing dataset, achieving higher MOS than GAN-Singer and FFT-Singer (3.85 vs. 3.74 and 3.67, respectively).
• Naive diffusion (without shallow diffusion) yields MOS of 3.71, while DiffSinger with shallow diffusion achieves 3.85.
• Shallow diffusion mechanism accelerates inference by 45.1% (RTF 0.191 vs. 0.348).
• A boundary predictor effectively selects k; using the predicted k yields better MOS than using a fixed or alternative k.
• Extensional experiments on LJSpeech show DiffSpeech achieving MOS improvements over FastSpeech 2 and Glow-TTS, with 29.2% speedup over naive diffusion.