[Paper Review] Speaking style adaptation in Text-To-Speech synthesis using Sequence-to-sequence models with attention

Bajibabu Bollepalli, Lauri Juvela+1 more|arXiv (Cornell University)|Oct 29, 2018

Speech Recognition and Synthesis27 references4 citations

TL;DR

This paper proposes a transfer learning approach to adapt a sequence-to-sequence text-to-speech (TTS) model trained on normal speech to generate high-quality Lombard-style speech using limited Lombard data. By fine-tuning the model with 30 minutes of Lombard speech and employing a WaveNet vocoder conditioned on mel-spectrograms, the system significantly outperforms baseline methods in both style similarity and naturalness, achieving state-of-the-art results in synthetic Lombard speech quality.

ABSTRACT

Currently, there are increasing interests in text-to-speech (TTS) synthesis to use sequence-to-sequence models with attention. These models are end-to-end meaning that they learn both co-articulation and duration properties directly from text and speech. Since these models are entirely data-driven, they need large amounts of data to generate synthetic speech with good quality. However, in challenging speaking styles, such as Lombard speech, it is difficult to record sufficiently large speech corpora. Therefore, in this study we propose a transfer learning method to adapt a sequence-to-sequence based TTS system of normal speaking style to Lombard style. Moreover, we experiment with a WaveNet vocoder in synthesis of Lombard speech. We conducted subjective evaluations to assess the performance of the adapted TTS systems. The subjective evaluation results indicated that an adaptation system with the WaveNet vocoder clearly outperformed the conventional deep neural network based TTS system in synthesis of Lombard speech.

Motivation & Objective

• To address the challenge of limited high-quality Lombard speech data for training end-to-end TTS systems.
• To improve speech intelligibility in noisy environments by enabling TTS systems to synthesize Lombard-style speech.
• To investigate whether transfer learning can effectively adapt a normal-style TTS model to Lombard style with minimal target-domain data.
• To evaluate the impact of different vocoders—particularly WaveNet—on the quality and naturalness of adapted Lombard speech.

Proposed method

• Fine-tune a pre-trained sequence-to-sequence TTS model (based on Tacotron) using only 30 minutes of Lombard speech data.
• Condition the WaveNet vocoder on mel-spectrograms extracted from the sequence-to-sequence model’s output to generate raw waveforms.
• Use transfer learning to adapt the model’s attention and decoder layers to learn Lombard-specific prosodic features such as increased F0 and vocal intensity.
• Apply speaker-conditional conditioning to preserve speaker identity while adapting speaking style.
• Conduct fine-tuning with both LSTM and Transformer-based encoder-decoder architectures to compare performance.
• Use a multi-stage training pipeline where the TTS model is first pre-trained on normal speech and then adapted to Lombard style.

Experimental results

Research questions

• RQ1 Can a sequence-to-sequence TTS model trained on normal speech be effectively adapted to generate high-quality Lombard-style speech with minimal target-domain data?
• RQ2 How does the choice of vocoder—specifically WaveNet versus traditional World vocoder—affect the naturalness and style accuracy of synthesized Lombard speech?
• RQ3 Does conditioning the WaveNet vocoder on mel-spectrograms improve the quality of adapted Lombard speech compared to end-to-end WaveNet training?
• RQ4 How does the performance of the adapted system compare to existing LSTM-based TTS systems in terms of style similarity and naturalness?
• RQ5 What is the impact of model architecture (LSTM vs. Transformer) on the effectiveness of style adaptation in low-data regimes?

Key findings

• System S5, which uses a WaveNet vocoder conditioned on mel-spectrograms, achieved the highest style similarity score, indicating it most closely matched the acoustic characteristics of natural Lombard speech.
• The WaveNet-based system (S5) significantly outperformed the baseline system (S1) in the CCR test for naturalness, with a statistically significant improvement in mean naturalness rating.
• Systems using the WaveNet vocoder consistently scored higher in both style similarity and naturalness evaluations than those using the World vocoder.
• The adaptation method successfully generated Lombard-like speech with only 30 minutes of target-domain data, demonstrating the effectiveness of transfer learning in low-resource settings.
• The Transformer-based model with WaveNet vocoder (S5) achieved the best overall performance, outperforming both LSTM-based and baseline systems in both subjective listening tests.
• The results confirm that conditioning the WaveNet vocoder on mel-spectrograms enhances the quality of synthesized Lombard speech, even with limited training data.