[论文解读] Initial investigation of an encoder-decoder end-to-end TTS framework using marginalization of monotonic hard latent alignments

Yusuke Yasuda, Xin Wang+1 more|arXiv (Cornell University)|Aug 30, 2019

Speech Recognition and Synthesis参考文献 27被引用 5

一句话总结

本文提出了一种新颖的端到端文本到语音（TTS）框架，用单调硬潜变量对齐替代软注意力机制，通过在网格结构上对这些对齐进行边缘化实现训练。该方法在无需复杂注意力模块或工程技巧的情况下实现了鲁棒的对齐学习，但主观评估显示语音质量较低（MOS 2.33），原因在于停顿时长被高估以及训练时间更长。

ABSTRACT

End-to-end text-to-speech (TTS) synthesis is a method that directly converts input text to output acoustic features using a single network. A recent advance of end-to-end TTS is due to a key technique called attention mechanisms, and all successful methods proposed so far have been based on soft attention mechanisms. However, although network structures are becoming increasingly complex, end-to-end TTS systems with soft attention mechanisms may still fail to learn and to predict accurate alignment between the input and output. This may be because the soft attention mechanisms are too flexible. Therefore, we propose an approach that has more explicit but natural constraints suitable for speech signals to make alignment learning and prediction of end-to-end TTS systems more robust. The proposed system, with the constrained alignment scheme borrowed from segment-to-segment neural transduction (SSNT), directly calculates the joint probability of acoustic features and alignment given an input text. The alignment is designed to be hard and monotonically increase by considering the speech nature, and it is treated as a latent variable and marginalized during training. During prediction, both the alignment and acoustic features can be generated from the probabilistic distributions. The advantages of our approach are that we can simplify many modules for the soft attention and that we can train the end-to-end TTS model using a single likelihood function. As far as we know, our approach is the first end-to-end TTS without a soft attention mechanism.

研究动机与目标

• 开发一种无需依赖复杂软注意力机制的更简单、更鲁棒的端到端TTS系统。
• 解决基于软注意力机制的TTS模型中常见的对齐错误，如跳过、重复和模糊。
• 用基于概率的原理性框架替代依赖工程技巧的组件，如停止标志预测和基于dropout的推理。
• 评估将硬性、单调对齐作为潜在变量并在训练中进行边缘化的可行性。
• 探究基于似然的训练目标是否能在无需软注意力机制的情况下实现具有竞争力的TTS性能。

提出的方法

• 模型采用序列到序列的编码器-解码器架构，将硬性、单调对齐视为潜在变量。
• 使用分段到序列神经转导（SSNT）框架计算声学特征与对齐的联合概率。
• 训练期间，通过输入词元和输出帧构成的网格上的前向-后向算法，对所有可能的对齐进行边缘化。
• 声学特征的发射概率建模为各向同性高斯分布，转移概率通过自回归神经网络学习。
• 推理阶段通过从学习到的联合分布中采样生成对齐和声学特征，直至达到最后一个输入词元为止。
• 整个模型通过单一似然目标函数端到端训练，避免了对辅助头或损失项的需求。

实验结果

研究问题

• RQ1 仅通过单调硬对齐的边缘化，能否实现无需软注意力机制的端到端TTS系统并达到具有竞争力的性能？
• RQ2 显式使用硬性单调对齐约束是否能减少跳过或重复等常见对齐错误？
• RQ3 基于似然的训练目标能否替代复杂的注意力机制以及停止标志预测等辅助组件？
• RQ4 该方法在标准朗读语音语料库上的性能与基于软注意力的TTS相比如何？
• RQ5 该方法的主要失败模式是什么？是否可通过模型改进或更长训练时间得到缓解？

主要发现

• 所提系统在标准朗读语音语料库上获得的平均意见得分（MOS）为2.33 ± 0.03，低于基线的软注意力模型。
• 低MOS的主要原因是停顿时长被高估，导致语音韵律不自然，尤其在包含停顿的句子中更为明显。
• 无停顿的句子MOS达到2.75 ± 0.06，表明模型在非停顿段落中表现更优。
• 由于硬对齐的固有单调性，模型成功避免了软注意力模型中常见的模糊、跳过和重复等对齐错误。
• 尽管对齐学习鲁棒，模型仍表现出新型对齐错误，如元音时长效长，表明发射概率建模存在局限性。
• 由于边缘化过程，训练速度显著变慢，表明更长的训练可能提升性能并减少错误。