[Paper Review] fairseq S2T: Fast Speech-to-Text Modeling with fairseq

Changhan Wang, Yun Tang+5 more|arXiv (Cornell University)|Oct 11, 2020

Natural Language Processing Techniques42 references92 citations

TL;DR

The paper introduces fairseq S2T, an extension of fairseq for end-to-end speech-to-text tasks, with scalable data processing, diverse model support (RNN/Transformer/Conformer), online/offline inference, and integration with MT/LM for multi-task learning.

ABSTRACT

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq's careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based, Transformer-based as well as Conformer-based models and open-source detailed training recipes. Fairseq's machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T documentation and examples are available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.

Motivation & Objective

• Motivate all-in-one S2S modeling for ASR and ST by leveraging large-scale unlabeled data and cross-task supervision (MT/LM).
• Provide end-to-end workflows from data pre-processing to training and inference in a scalable, extensible framework.
• Offer state-of-the-art model support (RNN, Transformer, Conformer) and detailed training recipes for reproducibility.
• Enable integration of fairseq MT models and LMs into S2T workflows for multi-task and transfer learning.

Proposed method

• Extend fairseq with S2T tasks including ASR and ST.
• Support RNN, Transformer, and Conformer architectures with CTC options for ASR.
• Provide online simultaneous ST policies (monotonic attention variants, wait-k, monotonic infinite lookback, monotonic multihead attention).
• Automate data preprocessing by extracting Kaldi-compliant features via PyKaldi or torchaudio, with optional pre-computation and ZIP packaging for I/O efficiency.
• Offer text data tokenization options (Moses, SentencePiece, subword-nmt, byte-level BPE, bytes) and YAML-based data configuration.
• Integrate evaluation metrics (WER, BLEU, chrF) and simultaneous ST metrics (AL, DAL) via simuleval, plus VizSeq visualization and TensorBoard monitoring.
• Leverage PyTorch-based scalable training (mixed precision, multi-GPU, multi-machine) and contribute ready-made training recipes.
• Allow seamless reuse of fairseq MT models and LMs for multi-task or transfer learning in S2T workflows.

Experimental results

Research questions

• RQ1 How does fairseq S2T perform on end-to-end ASR and ST benchmarks (e.g., LibriSpeech, MuST-C, CoVoST 2) using different backbone models?
• RQ2 Can multilingual training and cross-task supervision improve S2T performance compared to bilingual or single-task setups?
• RQ3 What is the impact of self-supervised features (SSL, e.g., wav2vec) on multilingual S2T performance and transferability?
• RQ4 How do online simultaneous ST models trade off latency (AL) and translation quality across language directions?
• RQ5 How does fairseq S2T compare with other toolkits in scalability, reproducibility, and integration with MT/LM models?

Key findings

• Fairseq S2T offers end-to-end S2T workflows with state-of-the-art RNN-, Transformer-, and Conformer-based models.
• Online simultaneous ST models with policies like wait-k and monotonic attention demonstrate competitive latency-accuracy trade-offs on MuST-C.
• Multilingual S2T models trained on multiple languages can outperform bilingual models on several directions, showing strong transferability.
• Transformer-based S2T models outperform RNN-based ones on CoVoST 2 across En–X and X–En directions, with gains from self-supervised features in low-resource and multilingual settings.
• CW-Lg (Conformer-based wav2vec) achieves competitive LibriSpeech WER, indicating strong ASR performance within the fairseq S2T framework.
• fairseq S2T integrates MT and LM components to enable multi-task learning and transfer learning, enhancing overall S2T performance.