[Paper Review] Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron

RJ Skerry-Ryan, Eric Battenberg+7 more|arXiv (Cornell University)|Mar 24, 2018

Speech Recognition and SynthesisComputer Science22 references219 citations

TL;DR

The paper extends Tacotron with a learned reference-prosody encoder to transfer prosody across utterances and speakers, enabling end-to-end prosody control and transfer. It evaluates with single- and multi-speaker Tacotron models using objective and subjective metrics.

ABSTRACT

We present an extension to the Tacotron speech synthesis architecture that learns a latent embedding space of prosody, derived from a reference acoustic representation containing the desired prosody. We show that conditioning Tacotron on this learned embedding space results in synthesized audio that matches the prosody of the reference signal with fine time detail even when the reference and synthesis speakers are different. Additionally, we show that a reference prosody embedding can be used to synthesize text that is different from that of the reference utterance. We define several quantitative and subjective metrics for evaluating prosody transfer, and report results with accompanying audio samples from single-speaker and 44-speaker Tacotron models on a prosody transfer task.

Motivation & Objective

• Motivate modeling prosody without explicit annotations by learning a latent prosody representation from acoustic signals.
• Extend Tacotron with a reference encoder to capture prosody and enable transfer between utterances and speakers.
• Evaluate the approach with quantitative metrics and human judgments across single- and multi-speaker setups.
• Investigate how prosody embeddings affect speaker identity and prosody transfer quality.

Proposed method

• Add a reference encoder that processes a reference acoustic signal into a fixed-length prosody embedding (dP=128, tanh activation) to condition Tacotron.
• Use a 6-layer convolutional reference encoder followed by a GRU to obtain the fixed-length prosody embedding.
• Condition the Tacotron decoder on text representations, speaker embeddings, and the prosody embedding via concatenation.
• Train end-to-end with Tacotron reconstruction loss, without explicit prosody supervision, to learn a compact prosody bottleneck.
• Optionally explore a sequence-based (variable-length) prosody representation with an additional attention head, though main results focus on fixed-length embeddings.

Experimental results

Research questions

• RQ1 Can a learned prosody embedding extracted from a reference audio reproduce and transfer prosody in Tacotron-generated speech?
• RQ2 Does conditioning Tacotron on a learned prosody space enable cross-speaker prosody transfer while preserving intelligibility and speaker identity?
• RQ3 How do prosody embedding size and activation affect transfer quality and information flow from reference to output?

Key findings

• Prosody transfer improves alignment to reference prosody across both single- and multi-speaker Tacotron models, compared to baselines without a reference encoder.
• Objective metrics (MCD and FFE) are substantially lower with the reference encoder than with the baseline across tested conditions.
• Subjective anchored prosody discrimination shows higher similarity to reference prosody when using the tanh-128 bottleneck than the baseline, across speaker configurations.
• Prosody can transfer when the reference speaker differs from the target, including unseen speakers, with notable effects on pitch and timing fidelity.
• Increasing bottleneck size generally improves prosody reproduction (MCD, FFE), while softmax bottlenecks underperform tanh-based bottlenecks.
• There is evidence of entanglement between prosody and speaker identity, with reference-prosody sometimes dominating perceived speaker traits.