[Paper Review] Fine-grained style control in Transformer-based Text-to-speech Synthesis

Liwei Chen, Alexander I. Rudnicky|arXiv (Cornell University)|Oct 12, 2021

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a novel Transformer-based text-to-speech system with fine-grained style control by introducing local style tokens (LST) extracted from reference speech via wav2vec 2.0 features. It uses cross-attention blocks to fuse content and style representations, reducing content leakage and improving naturalness, intelligibility, and style transferability—achieving state-of-the-art performance on LJ Speech and VCTK datasets without additional regularization.

ABSTRACT

In this paper, we present a novel architecture to realize fine-grained style control on the transformer-based text-to-speech synthesis (TransformerTTS). Specifically, we model the speaking style by extracting a time sequence of local style tokens (LST) from the reference speech. The existing content encoder in TransformerTTS is then replaced by our designed cross-attention blocks for fusion and alignment between content and style. As the fusion is performed along with the skip connection, our cross-attention block provides a good inductive bias to gradually infuse the phoneme representation with a given style. Additionally, we prevent the style embedding from encoding linguistic content by randomly truncating LST during training and using wav2vec 2.0 features. Experiments show that with fine-grained style control, our system performs better in terms of naturalness, intelligibility, and style transferability. Our code and samples are publicly available.

Motivation & Objective

• To enable fine-grained control over speaking styles such as prosody, speaking rate, and loudness in Transformer-based text-to-speech synthesis.
• To address content-leakage in style modeling, where linguistic content inadvertently leaks into style embeddings during training.
• To design a style representation that captures time-varying, frame-level prosodic variations without relying on global style embeddings alone.
• To improve speech quality and style transferability in TTS by fusing content and style representations through a cross-attention mechanism with skip connections.
• To demonstrate that the system generalizes across multi-speaker settings and enables zero-shot style transfer without emotion or style labels.

Proposed method

• The style network uses wav2vec 2.0 as a feature extractor from reference speech, providing robust, disentangled representations that reduce content leakage.
• Local style tokens (LST) are learned via a learnable codebook and multi-head attention, producing a sequence of frame-level style embeddings.
• A cross-attention block fuses phoneme representations with LST embeddings, enabling gradual infusion of style through skip connections and inductive bias.
• Content leakage is mitigated by randomly truncating LST during training and using wav2vec 2.0 features instead of raw waveforms or spectrograms.
• The system employs a content network with only phoneme embedding and layer normalization, and a decoder network similar to MultiSpeech.
• Style control is achieved by conditioning the decoder on both content and style representations, allowing independent manipulation of speaking style.

Fig. 1 : Overview of our LST-TTS system.

Experimental results

Research questions

• RQ1 Can fine-grained, frame-level speaking style control be effectively integrated into a Transformer-based TTS system without introducing content leakage?
• RQ2 How does the proposed cross-attention fusion mechanism between content and style representations affect speech naturalness and intelligibility?
• RQ3 To what extent can the system transfer subtle prosodic variations such as speaking rate and loudness from reference speech to synthesized speech?
• RQ4 Does the use of wav2vec 2.0 features as a representation backbone reduce content leakage compared to traditional spectral or embedding-based inputs?
• RQ5 Can the system generalize to multi-speaker settings and maintain high-quality style transfer without explicit emotion or style annotations?

Key findings

• The LST-A variant achieved a MOS of 3.62 ± 0.09 for naturalness and 3.28 ± 0.10 for style similarity on LJ Speech, outperforming prior methods including Tan et al. [10] on both metrics.
• On VCTK, the system maintained high performance with a MOS of 3.75 ± 0.10 for naturalness and 3.25 ± 0.12 for style similarity under the LST-A setup.
• The system achieved 68.4% accuracy in transferring emotion labels using a pretrained SER classifier, demonstrating effective emotion pattern transfer without any emotion supervision.
• Random truncation of LST during training (LST-R) did not degrade speech intelligibility, confirming that style control is robust and independent of sequential dependencies.
• The SPK variant showed higher WER due to content representation being adjusted to include prosodic information, suggesting improved robustness in alignment learning.
• The system generalizes well to multi-speaker settings, with both LST-A and LST-S achieving low WER on VCTK and maintaining high style similarity even when speaker identities differ between content and style inputs.