[论文解读] Causal Prosody Mediation for Text-to-Speech:Counterfactual Training of Duration, Pitch, and Energy in FastSpeech2

Suvendu Sekhar Mohanty|arXiv (Cornell University)|Mar 12, 2026

Emotion and Mood Recognition被引用 0

一句话总结

该论文提出一个因果韵律中介框架，在 FastSpeech2 上加入情感条件和反事实训练，以在语言内容与情感之间实现解耦，从而提升表达性 TTS 音韵与可控性。

ABSTRACT

We propose a novel causal prosody mediation framework for expressive text-to-speech (TTS) synthesis. Our approach augments the FastSpeech2 architecture with explicit emotion conditioning and introduces counterfactual training objectives to disentangle emotional prosody from linguistic content. By formulating a structural causal model of how text (content), emotion, and speaker jointly influence prosody (duration, pitch, energy) and ultimately the speech waveform, we derive two complementary loss terms: an Indirect Path Constraint (IPC) to enforce that emotion affects speech only through prosody, and a Counterfactual Prosody Constraint (CPC) to encourage distinct prosody patterns for different emotions. The resulting model is trained on multi-speaker emotional corpora (LibriTTS, EmoV-DB, VCTK) with a combined objective that includes standard spectrogram reconstruction and variance prediction losses alongside our causal losses. In evaluations on expressive speech synthesis, our method achieves significantly improved prosody manipulation and emotion rendering, with higher mean opinion scores (MOS) and emotion accuracy than baseline FastSpeech2 variants. We also observe better intelligibility (low WER) and speaker consistency when transferring emotions across speakers. Extensive ablations confirm that the causal objectives successfully separate prosody attribution, yielding an interpretable model that allows controlled counterfactual prosody editing (e.g. "same utterance, different emotion") without compromising naturalness. We discuss the implications for identifiability in prosody modeling and outline limitations such as the assumption that emotion effects are fully captured by pitch, duration, and energy. Our work demonstrates how integrating causal learning principles into TTS can improve controllability and expressiveness in generated speech.

研究动机与目标

• 通过在韵律中解耦情感与语言内容来推动表达性 TTS。
• 建立一个结构化因果模型，将文本、情感、说话人和韵律联系到语音。
• 引入能够强制因果约束并实现反事实韵律编辑的损失项。

提出的方法

• 在 FastSpeech2 上增加显式情感条件。
• 建立一个结构化因果模型，其中文本、情感和说话人影响韵律（时长、音高、能量）和波形。
• 推导两个损失项：Indirect Path Constraint (IPC) 和 Counterfactual Prosody Constraint (CPC)。
• 在多说话人情感语料库（LibriTTS、EmoV-DB、VCTK）上进行训练，结合标准声谱图与方差损失以及因果损失。
• 通过消融研究评估韵律操作、情感呈现、可懂度（WER）以及说话人一致性。

实验结果

研究问题

• RQ1 在因果框架下，情感对韵律的影响是否能完全由时长、音高和能量来捕捉？
• RQ2 IPC 与 CPC 约束是否能够在保持自然度与可懂度的前提下实现情感特定的韵律？
• RQ3 同一句话、不同情感的反事实编辑是否能在不损失性能的前提下实现感知上显著的韵律差异？
• RQ4 在跨说话人情感转移中，模型在 MOS、情感准确性和 WER 方面的表现如何？

主要发现

• 所提出的因果目标相较基线 FastSpeech2 变体提升了韵律操作性与情感呈现能力。
• 模型在 MOS 与情感准确性方面高于基线。
• 可懂度仍然较高（WER 低），在跨说话人进行情感转移时说话人一致性有所提升。
• 消融结果表明因果损失能够有效分离韵律归因并实现可解释的反事实编辑。
• 该方法讨论了可识别性与局限性，如情感效应可通过音高、时长和能量来捕捉。