[论文解读] FastS2S-VC: Streaming Non-Autoregressive Sequence-to-Sequence Voice Conversion

Hirokazu Kameoka, Kou Tanaka+1 more|arXiv (Cornell University)|Apr 14, 2021

Speech Recognition and Synthesis参考文献 58被引用 8

一句话总结

该论文提出 FastS2S-VC，一种非自回归序列到序列语音转换模型，通过教师-学生蒸馏框架将推理速度提升 70–100 倍，利用卷积注意力预测器并行生成注意力图，无需目标语音输入。该方法在 GPU 上实现仅 32 ms 的延迟，达到实时推理，同时保持高音频质量及与目标语音的相似性。

ABSTRACT

This paper proposes a non-autoregressive extension of our previously proposed sequence-to-sequence (S2S) model-based voice conversion (VC) methods. S2S model-based VC methods have attracted particular attention in recent years for their flexibility in converting not only the voice identity but also the pitch contour and local duration of input speech, thanks to the ability of the encoder-decoder architecture with the attention mechanism. However, one of the obstacles to making these methods work in real-time is the autoregressive (AR) structure. To overcome this obstacle, we develop a method to obtain a model that is free from an AR structure and behaves similarly to the original S2S models, based on a teacher-student learning framework. In our method, called "FastS2S-VC", the student model consists of encoder, decoder, and attention predictor. The attention predictor learns to predict attention distributions solely from source speech along with a target class index with the guidance of those predicted by the teacher model from both source and target speech. Thanks to this structure, the model is freed from an AR structure and allows for parallelization. Furthermore, we show that FastS2S-VC is suitable for real-time implementation based on a sliding-window approach, and describe how to make it run in real-time. Through speaker-identity and emotional-expression conversion experiments, we confirmed that FastS2S-VC was able to speed up the conversion process by 70 to 100 times compared to the original AR-type S2S-VC methods, without significantly degrading the audio quality and similarity to target speech. We also confirmed that the real-time version of FastS2S-VC can be run with a latency of 32 ms when run on a GPU.

研究动机与目标

• 解决自回归序列到序列语音转换（S2S-VC）模型在实时应用中的高延迟问题。
• 开发一种非自回归（NAR）S2S-VC 变体，在保持性能的同时实现并行推理。
• 通过滑动窗口推理策略实现实时部署，实现低延迟。
• 在移除自回归结构的情况下，仍保持高音频质量及说话人/情感表达相似性。

提出的方法

• 该模型采用教师-学生蒸馏框架，学生模型学习模仿教师 S2S-VC 模型的注意力分布。
• 注意力预测器为全卷积网络，仅基于源梅尔谱图和目标类别索引，利用受限高斯函数并行生成注意力权重矩阵。
• 学生模型在推理过程中不输入目标梅尔谱图，消除自回归依赖，实现并行生成。
• 模型架构包括预网络（prenet）、编码器、注意力预测器、预解码器、后解码器和后网络，用于端到端特征映射与波形生成。
• 通过滑动窗口推理方法实现实时运行，将输入语音分块处理，延迟最小化。
• 波形生成采用 Parallel WaveGAN，所有方法均一致使用以保证公平评估。

实验结果

研究问题

• RQ1 非自回归 S2S-VC 模型能否在实现并行推理的同时，达到与自回归 S2S-VC 相当的语音转换质量？
• RQ2 从自回归教师模型蒸馏知识，在非自回归学生模型中实现快速并行推理的效果如何？
• RQ3 在 GPU 和 CPU 上结合实时滑动窗口处理时，该方法可实现的延迟是多少？
• RQ4 在音频质量、说话人相似性及情感表达相似性方面，非自回归模型与自回归 S2S-VC 的性能相比如何？

主要发现

• FastS2S-VC 相较原始自回归 S2S-VC 模型，语音转换处理速度提升了 70 至 100 倍。
• FastS2S-VC–RT 在 GPU 上执行时，延迟仅为 32 ms，支持实际的实时部署。
• 在 CPU 上，可行延迟为 256 ms，表明 GPU 加速对低延迟运行至关重要。
• 主观 MOS 测试显示，FastS2S-VC–RT–Conv 的音频质量评分较高，优于基于 RNN 的自回归 S2S-VC，且接近性能最佳的 S2S-VC–BAT–Conv 模型。
• FastS2S-VC 的说话人相似性与情感表达相似性得分与自回归 S2S-VC–BAT–RNN 基线相当或更优。
• 在客观评估中，FastS2S-VC 的卷积版本优于基于 Transformer 的版本，因此被选为实时实验的代表性模型。