[Paper Review] HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis

Jiawei Chen, Xu Tan+3 more|arXiv (Cornell University)|Sep 3, 2020

Speech Recognition and Synthesis36 references43 citations

TL;DR

HiFiSinger uses FastSpeech acoustic modeling and Parallel WaveGAN vocoding with multi-scale adversarial training (SF-GAN and ML-GAN) to achieve high-fidelity 48kHz singing voice synthesis, outperforming prior SVS systems.

ABSTRACT

High-fidelity singing voices usually require higher sampling rate (e.g., 48kHz) to convey expression and emotion. However, higher sampling rate causes the wider frequency band and longer waveform sequences and throws challenges for singing voice synthesis (SVS) in both frequency and time domains. Conventional SVS systems that adopt small sampling rate cannot well address the above challenges. In this paper, we develop HiFiSinger, an SVS system towards high-fidelity singing voice. HiFiSinger consists of a FastSpeech based acoustic model and a Parallel WaveGAN based vocoder to ensure fast training and inference and also high voice quality. To tackle the difficulty of singing modeling caused by high sampling rate (wider frequency band and longer waveform), we introduce multi-scale adversarial training in both the acoustic model and vocoder to improve singing modeling. Specifically, 1) To handle the larger range of frequencies caused by higher sampling rate, we propose a novel sub-frequency GAN (SF-GAN) on mel-spectrogram generation, which splits the full 80-dimensional mel-frequency into multiple sub-bands and models each sub-band with a separate discriminator. 2) To model longer waveform sequences caused by higher sampling rate, we propose a multi-length GAN (ML-GAN) for waveform generation to model different lengths of waveform sequences with separate discriminators. 3) We also introduce several additional designs and findings in HiFiSinger that are crucial for high-fidelity voices, such as adding F0 (pitch) and V/UV (voiced/unvoiced flag) as acoustic features, choosing an appropriate window/hop size for mel-spectrogram, and increasing the receptive field in vocoder for long vowel modeling. Experiment results show that HiFiSinger synthesizes high-fidelity singing voices with much higher quality: 0.32/0.44 MOS gain over 48kHz/24kHz baseline and 0.83 MOS gain over previous SVS systems.

Motivation & Objective

• Address the challenge of high-fidelity singing voice synthesis at 48kHz by expanding frequency coverage and temporal modeling.
• Propose adversarial training regimes to model wide frequency bands and long waveform sequences.
• Incorporate pitch and voiced/unvoiced features to improve vocal quality and controllability.
• Investigate design choices (window size, receptive field) that affect high-frequency singing voice synthesis.
• Demonstrate substantial quality gains over 24kHz baselines and previous SVS systems.

Proposed method

• Adopt a FastSpeech-based acoustic model and a Parallel WaveGAN vocoder for fast training and inference.
• Introduce SF-GAN: split 80 mel-spectrogram bins into low/mid/high sub-bands with separate discriminators to better model wide frequency bands.
• Introduce ML-GAN: use multiple discriminators over different waveform lengths (0.25s, 0.5s, 0.75s, 1.0s) to handle longer sequences.
• Incorporate pitch (F0) and V/UV as additional acoustic features; use the music score pitch as a shortcut input for residual learning.
• Choose 20ms window and 5ms hop for mel-spectrograms and enlarge vocoder receptive field to model long vowels.
• Train acoustic model and vocoder separately; apply discriminators after warm-up; use ground-truth durations during training and predicted ones during inference.

Experimental results

Research questions

• RQ1 Can 48kHz SVS deliver significantly higher quality than 24kHz SVS and prior SVS systems?
• RQ2 Do multi-band (SF-GAN) and multi-length (ML-GAN) adversarial designs improve mel-spectrogram and waveform generation for high-sampling-rate singing?
• RQ3 What is the impact of additional acoustic features (F0, V/UV) and careful feature/window choices on final voice quality?
• RQ4 How does the receptive field of the vocoder affect long vowels in singing voices?

Key findings

• HiFiSinger at 48kHz achieves higher MOS than both 24kHz baselines and previous SVS systems, including a 0.83 MOS gain over XiaoiceSing.
• HiFiSinger shows 0.29 MOS gain when upgrading from 24kHz to 48kHz, outperforming the baseline gain at 0.12 MOS.
• SF-GAN with three frequency-band discriminators outperforms configurations with fewer or more discriminators, indicating a sweet spot for modeling across bands.
• ML-GAN with multiple waveform-length discriminators reduces glitches and vibrations in long vowels compared to single-length discriminators.
• Pitch and V/UV inputs significantly improve vocoder quality, with CMOS drops when removed (−0.28 to −0.34) demonstrating their importance.
• An appropriately chosen 20ms/5ms mel-spectrogram windowing and a larger vocoder receptive field contribute to higher fidelity singing synthesis.