[论文解读] HiFiSinger: Towards High-Fidelity Neural Singing Voice Synthesis

Jiawei Chen, Xu Tan+3 more|arXiv (Cornell University)|Sep 3, 2020

Speech Recognition and Synthesis参考文献 36被引用 43

一句话总结

HiFiSinger 使用 FastSpeech 声学建模和 Parallel WaveGAN 的声码器，结合多尺度对抗训练（SF-GAN 和 ML-GAN）实现高保真度的 48kHz 唱歌语音合成，优于现有的 SVS 系统。

ABSTRACT

High-fidelity singing voices usually require higher sampling rate (e.g., 48kHz) to convey expression and emotion. However, higher sampling rate causes the wider frequency band and longer waveform sequences and throws challenges for singing voice synthesis (SVS) in both frequency and time domains. Conventional SVS systems that adopt small sampling rate cannot well address the above challenges. In this paper, we develop HiFiSinger, an SVS system towards high-fidelity singing voice. HiFiSinger consists of a FastSpeech based acoustic model and a Parallel WaveGAN based vocoder to ensure fast training and inference and also high voice quality. To tackle the difficulty of singing modeling caused by high sampling rate (wider frequency band and longer waveform), we introduce multi-scale adversarial training in both the acoustic model and vocoder to improve singing modeling. Specifically, 1) To handle the larger range of frequencies caused by higher sampling rate, we propose a novel sub-frequency GAN (SF-GAN) on mel-spectrogram generation, which splits the full 80-dimensional mel-frequency into multiple sub-bands and models each sub-band with a separate discriminator. 2) To model longer waveform sequences caused by higher sampling rate, we propose a multi-length GAN (ML-GAN) for waveform generation to model different lengths of waveform sequences with separate discriminators. 3) We also introduce several additional designs and findings in HiFiSinger that are crucial for high-fidelity voices, such as adding F0 (pitch) and V/UV (voiced/unvoiced flag) as acoustic features, choosing an appropriate window/hop size for mel-spectrogram, and increasing the receptive field in vocoder for long vowel modeling. Experiment results show that HiFiSinger synthesizes high-fidelity singing voices with much higher quality: 0.32/0.44 MOS gain over 48kHz/24kHz baseline and 0.83 MOS gain over previous SVS systems.

研究动机与目标

• 解决 48kHz 高保真唱歌语音合成的挑战，通过扩展频率覆盖和时间建模。
• 提出对抗性训练方案以建模宽频带和长波形序列。
• 将音高和有/无声特征融入以提升发声质量和可控性。
• 研究影响高频唱歌语音合成的设计选择（窗口大小、感受野）。
• 证明相对于 24kHz 基线和以往 SVS 系统的显著质量提升。

提出的方法

• 采用基于 FastSpeech 的声学模型和 Parallel WaveGAN 声码器以实现快速训练和推理。
• 引入 SF-GAN：将 80 个梅尔频谱箱分为低/中/高子带，并使用独立判别器以更好地建模宽频带。
• 引入 ML-GAN：在不同波形长度（0.25s、0.5s、0.75s、1.0s）上使用多个判别器以处理更长的序列。
• 将音高（F0）和 V/UV 作为额外声学特征；使用乐谱音高作为残差学习的捷径输入。
• 为梅尔谱选择 20ms 窗口和 5ms 跳步，并扩大声码器感受野以建模长元音。
• 对声学模型和声码器分开训练； warm-up 后应用判别器；训练阶段使用真实时长，推理阶段使用预测时长。

实验结果

研究问题

• RQ1 48kHz SVS 是否能显著高于 24kHz SVS 和以往的 SVS 系统？
• RQ2 多带（SF-GAN）和多长度（ML-GAN）对抗设计是否提升高采样率唱歌的梅尔谱和波形生成？
• RQ3 额外声学特征（F0、V/UV）和谨慎的特征/窗口选择对最终发声质量有何影响？
• RQ4 声码器的感受野如何影响唱歌中的长元音？

主要发现

• HiFiSinger 在 48kHz 下的 MOS 高于 24kHz 基线和以往的 SVS 系统，包括比 XiaoiceSing 的 MOS 提升 0.83。
• 从 24kHz 升级到 48kHz 时，HiFiSinger 的 MOS 增益为 0.29，超出基线在 0.12 的增益。
• 拥有三个频带判别器的 SF-GAN 在不同数量的判别器配置中表现最佳，显示出对跨带建模的最佳平衡点。
• 使用多波形长度判别器的 ML-GAN 相较单一长度判别器，减少了长元音中的错位与颤动。
• 音高与 V/UV 输入显著提升声码器质量，移除时的 CMOS 降分（−0.28 至 −0.34）表明其重要性。
• 适当选择的 20ms/5ms 梅尔谱窗口和更大的声码器感受野有助于实现更高保真度的唱歌合成。