[Paper Review] Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesis

Yanyao Bian, Changbin Chen+2 more|arXiv (Cornell University)|Apr 4, 2019

Speech Recognition and Synthesis14 references42 citations

TL;DR

Introduces a multi-reference encoder for Tacotron with intercross training to disentangle, transfer, and independently control multiple speech style classes. Demonstrates independent control of speaker, emotion, prosody, and other styles through separate sub-encoders.

ABSTRACT

Speech style control and transfer techniques aim to enrich the diversity and expressiveness of synthesized speech. Existing approaches model all speech styles into one representation, lacking the ability to control a specific speech feature independently. To address this issue, we introduce a novel multi-reference structure to Tacotron and propose intercross training approach, which together ensure that each sub-encoder of the multi-reference encoder independently disentangles and controls a specific style. Experimental results show that our model is able to control and transfer desired speech styles individually.

Motivation & Objective

• Motivate expressive TTS by enabling independent control of multiple speech style classes.
• Propose a multi-reference encoder structure to model distinct style classes separately.
• Introduce intercross training to enforce disentanglement across style encoders.
• Add auxiliary tasks to stabilize training and encourage orthogonality between style embeddings.
• Demonstrate style disentangling, transfer, and control experimentally across single- and multi-reference setups.

Proposed method

• Extend GST-Tacotron with a multi-reference encoder consisting of N GST sub-encoders.
• Each sub-encoder uses a reference encoder and multi-head attention to produce a style embedding.
• Enforce disentanglement by ensuring each sub-encoder represents a specific style class (Equation 1).
• Apply intercross training (IT) to exchange posteriors during reconstruction (Equations 2-5).
• Incorporate optional auxiliary tasks: style classification loss and an orthogonality loss to encourage independent style spaces (Equation 7).
• Provide inference procedures for style disentangling, transfer, and interpolation between style embeddings (Equations 8-9).

Experimental results

Research questions

• RQ1 Can a multi-reference encoder disentangle multiple speech style classes into independent representations?
• RQ2 Does intercross training enable robust style disentangling, transfer, and independent control across styles?
• RQ3 Can style embeddings be interpolated to smoothly control attributes like speaker and prosody?
• RQ4 Is it feasible to perform few-shot or one-shot style transfer with new speakers using the multi-reference approach?
• RQ5 Do auxiliary tasks (classification and orthogonality) stabilize training and improve disentanglement?

Key findings

• The multi-reference model produces style embeddings that cluster by the intended style class, indicating effective disentanglement.
• Intercross training enables non-parallel style transfer with preserved text length and naturalness, improving transfer quality over single-reference baselines.
• Style embeddings support smooth linear interpolation, enabling independent control of speaker and prosody features.
• Embeddings from different sub-encoders are clustered by their respective style classes, demonstrating independent control in multi-reference setups.
• The approach supports few-shot and one-shot speaker conversion, with fine-tuning able to achieve high accept rates on new speakers.