[论文解读] FastSpeech 2: Fast and High-Quality End-to-End Text to Speech

Yi Ren|arXiv (Cornell University)|Jun 8, 2020

Speech Recognition and Synthesis参考文献 40被引用 513

一句话总结

FastSpeech 2 直接在 ground-truth mel-spectrograms 上训练，并加入音高和能量方差信息以缓解一对多映射；FastSpeech 2s 进一步在并行中直接生成波形，以实现更快的端到端 TTS。

ABSTRACT

Non-autoregressive text to speech (TTS) models such as FastSpeech can synthesize speech significantly faster than previous autoregressive models with comparable quality. The training of FastSpeech model relies on an autoregressive teacher model for duration prediction (to provide more information as input) and knowledge distillation (to simplify the data distribution in output), which can ease the one-to-many mapping problem (i.e., multiple speech variations correspond to the same text) in TTS. However, FastSpeech has several disadvantages: 1) the teacher-student distillation pipeline is complicated and time-consuming, 2) the duration extracted from the teacher model is not accurate enough, and the target mel-spectrograms distilled from teacher model suffer from information loss due to data simplification, both of which limit the voice quality. In this paper, we propose FastSpeech 2, which addresses the issues in FastSpeech and better solves the one-to-many mapping problem in TTS by 1) directly training the model with ground-truth target instead of the simplified output from teacher, and 2) introducing more variation information of speech (e.g., pitch, energy and more accurate duration) as conditional inputs. Specifically, we extract duration, pitch and energy from speech waveform and directly take them as conditional inputs in training and use predicted values in inference. We further design FastSpeech 2s, which is the first attempt to directly generate speech waveform from text in parallel, enjoying the benefit of fully end-to-end inference. Experimental results show that 1) FastSpeech 2 achieves a 3x training speed-up over FastSpeech, and FastSpeech 2s enjoys even faster inference speed; 2) FastSpeech 2 and 2s outperform FastSpeech in voice quality, and FastSpeech 2 can even surpass autoregressive models. Audio samples are available at https://speechresearch.github.io/fastspeech2/.

研究动机与目标

• 通过用方差信息（时长、音高、能量）丰富输入，解决非自回归 TTS 的一对多映射问题。
• 通过直接使用 ground-truth mel-spectrograms 而不是教师蒸馏目标来简化训练流程。
• 提高时长准确性并引入音高与能量预测器以提升语音质量。
• 开发 FastSpeech 2s，使得能够进行完全端到端文本到波形合成并实现快速推理。

提出的方法

• 编码器将音素嵌入转换为隐藏表示。
• 方差适配器向隐藏序列添加时长、音高和能量信息。
• 时长预测器使用强制对齐（MFA）目标以获得准确的音素时长。
• 音高预测器通过连续小波变换（CWT）在频域预测音高，并使用 iCWT 进行推断。
• 能量预测器预测帧级能量并将其嵌入添加到隐藏序列中。
• 梅尔-光谱解码器并行生成梅尔-光谱；训练目标为 ground-truth mel-spectrograms（无教师-学生蒸馏）。
• FastSpeech 2s 通过直接从文本生成波形，使用带对抗训练的 WaveNet 风格解码器和并行的声码器判别器扩展了 FastSpeech 2；实现端到端波形合成。

实验结果

研究问题

• RQ1 去掉教师-学生蒸馏是否通过直接在 ground-truth mel-spectrograms 上训练来提升语音质量？
• RQ2 将显式的时长、音高（通过 CWT）和能量作为方差信息纳入，是否能提升合成语音的自然度？
• RQ3 在频域预测音高对韵律和MOS有何影响？
• RQ4 在保持质量与速度的前提下，FastSpeech 2s 是否能实现完全端到端的非自回归 TTS 波形直接生成？

主要发现

• FastSpeech 2 相较于 FastSpeech 提升了约 3 倍的训练速度。
• FastSpeech 2 与 2s 在语音质量方面优于 FastSpeech；FastSpeech 2 的 MOS 甚至超过自回归模型。
• FastSpeech 2 与 2s 提供更准确的音高和能量，比 FastSpeech 提升韵律与 CMOS 得分。
• 音高通过在频域使用 CWT 预测，相较时域预测在音高建模和 CMOS 得分上有所提升。
• FastSpeech 2s 实现了完全端到端的波形生成，推理速度比级联系统更快。
• FastSpeech 2 在推理端提供显著的速度提升（RTF 约为 1.95e-2），FastSpeech 2s 进一步改善端点延迟。