[Paper Review] fairseq S^2: A Scalable and Integrable Speech Synthesis Toolkit

Changhan Wang, Wei-Ning Hsu+6 more|arXiv (Cornell University)|Sep 14, 2021

Speech Recognition and Synthesis61 references4 citations

TL;DR

This paper introduces fairseq S², a scalable and integrable speech synthesis toolkit built as an extension to the fairseq framework, enabling end-to-end training of autoregressive and non-autoregressive text-to-speech models with support for multi-speaker synthesis, preprocessing tools for low-resource data, and automatic evaluation metrics. It demonstrates improved model performance and generalization using minimal-curated data through VAD, noise filtering, and pre-trained speaker embeddings.

ABSTRACT

This paper presents fairseq S^2, a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built and their importance is shown empirically. To facilitate faster iteration of development and analysis, a suite of automatic metrics is included. Apart from the features added specifically for this extension, fairseq S^2 also benefits from the scalability offered by fairseq and can be easily integrated with other state-of-the-art systems provided in this framework. The code, documentation, and pre-trained models are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_synthesis.

Motivation & Objective

• To address the lack of unified, scalable toolkits for speech synthesis that support diverse model architectures and data types.
• To enable training high-quality TTS models on less curated, in-the-wild speech data through robust preprocessing pipelines.
• To facilitate faster development and evaluation by integrating automatic metrics and pre-trained components.
• To support integration with other state-of-the-art models in fairseq, including self-supervised speech representations and unit-to-speech systems.
• To enable zero-shot speaker synthesis using pre-trained speaker embeddings, extending generalization to unseen speakers.

Proposed method

• Extends fairseq with a suite of text-to-spectrogram models, including Tacotron2, Transformer, and FastSpeech2, supporting both autoregressive and non-autoregressive inference.
• Introduces multi-speaker capabilities via learnable speaker embedding lookup tables and pre-trained speaker embeddings from verification models.
• Implements audio preprocessing pipelines including voice activity detection (VAD), noise reduction (DN), and filtering based on SNR and CER for data curation.
• Integrates automatic evaluation metrics such as MCD, CER, MOS, and F0/RMS correlation to accelerate model iteration.
• Supports both text and learned units (from self-supervised models) as input, enabling text-free TTS systems.
• Enables seamless integration with existing fairseq models for ASR, MT, LM, and self-supervised learning via shared codebase and modular design.

Experimental results

Research questions

• RQ1 Can preprocessing pipelines like VAD, noise reduction, and filtering improve training stability and performance on low-resource, in-the-wild speech data?
• RQ2 How does using pre-trained speaker embeddings compare to learnable embeddings in terms of zero-shot speaker synthesis and model performance?
• RQ3 To what extent can automatic metrics like MCD and CER accelerate model development and reduce reliance on subjective human evaluation?
• RQ4 Can fairseq S² effectively support end-to-end unit-to-speech synthesis using self-supervised speech representations as input?
• RQ5 How does the reduction factor in autoregressive models affect training convergence and synthesis quality on noisy or imperfect data?

Key findings

• Using VAD with aggressiveness level 3 reduced silence by 40% on average, but risked truncating speech; optimal settings balanced silence removal and speech preservation.
• Training with DN+VAD-3+FLT preprocessing reduced CER to below 10% and improved MOS by 0.2–0.3 points compared to raw audio, especially on VCTK and Common Voice.
• A reduction factor of 4 in the Transformer TTS model yielded better performance than 2, with training failing entirely at reduction factor 2 on raw audio without preprocessing.
• Pre-trained speaker embeddings achieved comparable performance to learnable embeddings while enabling zero-shot synthesis for unseen speakers.
• Filtering data by SNR and CER reduced MCD by 0.1 and CER by 1.5, indicating improved model fitting and intelligibility on challenging examples.
• The integration of fairseq S² with self-supervised models enabled successful unit-to-speech synthesis, demonstrating feasibility of text-free TTS pipelines.