[论文解读] Neural source-filter waveform models for statistical parametric speech synthesis

Xin Wang, Shinji Takaki+1 more|arXiv (Cornell University)|Apr 27, 2019

Speech Recognition and Synthesis参考文献 53被引用 8

一句话总结

该论文提出了一种神经源-滤波器（NSF）框架，用于统计参数化语音合成，通过正弦激励源、扩张卷积滤波器和条件特征处理，实现比WaveNet快100倍的高质量语音波形生成，采用STFT最小化频谱幅度差异进行端到端训练，实现非自回归、一次生成的WaveNet级语音质量。

ABSTRACT

Neural waveform models such as WaveNet have demonstrated better performance than conventional vocoders for statistical parametric speech synthesis. As an autoregressive (AR) model, WaveNet is limited by a slow sequential waveform generation process. Some new models that use the inverse-autoregressive flow (IAF) can generate a whole waveform in a one-shot manner. However, these IAF-based models require sequential transformation during training, which severely slows down the training speed. Other models such as Parallel WaveNet and ClariNet bring together the benefits of AR and IAF-based models and train an IAF model by transferring the knowledge from a pre-trained AR teacher to an IAF student without any sequential transformation. However, both models require additional training criteria, and their implementation is prohibitively complicated. We propose a framework for neural source-filter (NSF) waveform modeling without AR nor IAF-based approaches. This framework requires only three components for waveform generation: a source module that generates a sine-based signal as excitation, a non-AR dilated-convolution-based filter module that transforms the excitation into a waveform, and a conditional module that pre-processes the acoustic features for the source and filer modules. This framework minimizes spectral-amplitude distances for model training, which can be efficiently implemented by using short-time Fourier transform routines. Under this framework, we designed three NSF models and compared them with WaveNet. It was demonstrated that the NSF models generated waveforms at least 100 times faster than WaveNet, and the quality of the synthetic speech from the best NSF model was better than or equally good as that from WaveNet.

研究动机与目标

• 解决如WaveNet等自回归（AR）神经声码器在统计参数化语音合成中推理速度慢的问题。
• 克服基于流（flow-based）或知识蒸馏方法（如Parallel WaveNet和ClariNet）的复杂性与训练开销。
• 为TTS开发一种简单、高效且高质量的AR与IAF-based波形模型替代方案。
• 实现在不牺牲感知质量的前提下，快速、一次生成波形。
• 探究正弦激励在简化语音波形中周期性和基频（F0）结构学习中的作用。

提出的方法

• NSF框架由三部分组成：生成正弦激励信号的激励模块，将激励转换为波形的非自回归扩张卷积滤波模块，以及为两个模块处理输入声学特征的条件模块。
• 通过短时傅里叶变换（STFT）计算生成波形与目标波形之间的频谱幅度距离，最小化该损失进行端到端训练。
• 提出三种NSF变体：基础NSF（b-NSF）、简化NSF（s-NSF）和谐波-噪声NSF（hn-NSF），其中hn-NSF对谐波与噪声分量分别使用独立的源-滤波器对。
• 通过STFT高效实现训练目标，实现无需序列生成或复杂流训练的快速优化。
• 滤波模块设计为带有跳跃连接的残差网络，实现激励到自然语音波形的渐进式转换。
• F0输入直接用于激励模块，实现生成输出中显式的音高控制。

实验结果

研究问题

• RQ1 非自回归神经源-滤波器框架能否在实现一次生成、快速波形生成的同时，达到WaveNet级的语音质量？
• RQ2 使用正弦激励信号如何简化神经声码器中对周期性和F0结构的学习？
• RQ3 架构简化以及对谐波与噪声分量的独立建模对波形质量与训练效率有何影响？
• RQ4 基于STFT的频谱幅度损失与其它训练目标相比，在感知质量与训练稳定性方面表现如何？
• RQ5 与AR和IAF-based方法相比，NSF框架在多大程度上减少了训练时间与模型复杂度？

主要发现

• hn-NSF模型在单说话人日语语料的大规模听音测试中，语音质量与WaveNet相当。
• 最佳NSF模型的波形生成速度至少比WaveNet声码器快100倍，支持实时推理。
• 输入F0与生成波形中提取的F0之间的相关性很高（hn-NSF中F0r3为0.988），表明音高跟踪准确。
• hn-NSF模型中的滤波模块成功将正弦激励转换为具有共振峰结构的波形，如各层谱图演化所示。
• 消融实验证实，滤波模块中的跳跃连接对性能至关重要，若移除正弦激励则性能严重下降。
• 基于STFT的频谱幅度损失可实现高效训练与稳定收敛，即使在多种STFT配置下亦成立。