[Paper Review] MultiSpeech: Multi-Speaker Text to Speech with Transformer

Mingjian Chen, Xu Tan+6 more|arXiv (Cornell University)|Jun 8, 2020

Speech Recognition and Synthesis18 references30 citations

TL;DR

MultiSpeech introduces a robust multi-speaker Transformer TTS with three techniques—diagonal attention constraint, encoder layer normalization on phoneme embeddings, and a small decoder pre-net bottleneck—to improve text-to-speech alignment and voice quality; it also serves as a teacher to train a fast, multi-speaker FastSpeech model.

ABSTRACT

Transformer-based text to speech (TTS) model (e.g., Transformer TTS~\cite{li2019neural}, FastSpeech~\cite{ren2019fastspeech}) has shown the advantages of training and inference efficiency over RNN-based model (e.g., Tacotron~\cite{shen2018natural}) due to its parallel computation in training and/or inference. However, the parallel computation increases the difficulty while learning the alignment between text and speech in Transformer, which is further magnified in the multi-speaker scenario with noisy data and diverse speakers, and hinders the applicability of Transformer for multi-speaker TTS. In this paper, we develop a robust and high-quality multi-speaker Transformer TTS system called MultiSpeech, with several specially designed components/techniques to improve text-to-speech alignment: 1) a diagonal constraint on the weight matrix of encoder-decoder attention in both training and inference; 2) layer normalization on phoneme embedding in encoder to better preserve position information; 3) a bottleneck in decoder pre-net to prevent copy between consecutive speech frames. Experiments on VCTK and LibriTTS multi-speaker datasets demonstrate the effectiveness of MultiSpeech: 1) it synthesizes more robust and better quality multi-speaker voice than naive Transformer based TTS; 2) with a MutiSpeech model as the teacher, we obtain a strong multi-speaker FastSpeech model with almost zero quality degradation while enjoying extremely fast inference speed.

Motivation & Objective

• Improve text-to-speech alignment in multi-speaker Transformer TTS under noisy, diverse speaker conditions.
• Propose architectural and training techniques to stabilize and enhance synthesis quality.
• Enable fast, scalable multi-speaker inference and enable knowledge distillation to FastSpeech.

Proposed method

• Introduce diagonal constraint on encoder-decoder attention to encourage monotonic, diagonal alignments.
• Apply layer normalization to encoder phoneme embeddings to preserve position information before adding positional embeddings.
• Use a small decoder pre-net bottleneck (e.g., 80-32-32-256) to prevent copying from adjacent frames and force text-driven prediction.
• Incorporate an attention sliding window during autoregressive inference to maintain alignment progression.
• Train MultiSpeech on multi-speaker datasets (VCTK, LibriTTS) with Transformer blocks (4 layers each, hidden size 256).
• Use WaveNet as vocoder for waveform synthesis and perform MOS-based evaluation; extend to FastSpeech via teacher-student distillation.

Figure 1: The model structure of our proposed MultiSpeech. The green blocks are the newly added modules for multi-speaker TTS based on Transformer.

Experimental results

Research questions

• RQ1 Can diagonal attention constraints improve alignment learning in multi-speaker Transformer TTS?
• RQ2 Does applying layer normalization to phoneme embeddings improve position information and alignment quality?
• RQ3 Does a pre-net bottleneck prevent frame copy and promote text-driven decoding in multi-speaker Transformer TTS?
• RQ4 Can MultiSpeech serve as an effective teacher for fast multi-speaker TTS (FastSpeech) without quality degradation?

Key findings

• MultiSpeech yields higher MOS than a naive Transformer-based multi-speaker TTS on VCTK and LibriTTS.
• On VCTK, MultiSpeech MOS = 3.65 (0.14), with diagonal rate r = 0.694; on LibriTTS MOS = 2.95 (0.14).
• Ablation shows removing diagonal constraint, layer normalization, or pre-net bottleneck lowers MOS and/or diagonal rate; removing all three reduces MOS to 2.64 (0.35) and r to 0.366.
• Layer normalization (LN) of encoder inputs provides higher diagonal attention rate (r = 0.694) than learnable weight (LW) and baseline (0.506 and 0.637 respectively).
• Using MultiSpeech as a teacher, a multi-speaker FastSpeech model achieves MOS 3.53 (0.22) versus GT 4.02 (0.09) and FastSpeech baseline 3.45 (0.13) on VCTK, with substantially faster inference.
• MultiSpeech MOS on GT mel + Vocoder approaches GT.

Figure 2: (a) The illustration of diagonal constraint in attention, where the above figure has a small diagonal constraint loss and the below figure has a large diagonal constraint loss. (b) The model structure of the pre-net bottleneck in decoder.