[Paper Review] Self-Training for End-to-End Speech Translation

Juan Pino, Qiantong Xu+3 more|arXiv (Cornell University)|Jun 3, 2020

Natural Language Processing Techniques28 references4 citations

TL;DR

This paper proposes a self-training framework for end-to-end speech translation that leverages pseudo-labels from unlabeled audio using either a cascade ASR+MT system or an end-to-end ST model. It achieves state-of-the-art results on MuST-C English-French and English-German, with 8.3 and 5.7 BLEU gains over a strong semi-supervised baseline, respectively.

ABSTRACT

One of the main challenges for end-to-end speech translation is data scarcity. We leverage pseudo-labels generated from unlabeled audio by a cascade and an end-to-end speech translation model. This provides 8.3 and 5.7 BLEU gains over a strong semi-supervised baseline on the MuST-C English-French and English-German datasets, reaching state-of-the art performance. The effect of the quality of the pseudo-labels is investigated. Our approach is shown to be more effective than simply pre-training the encoder on the speech recognition task. Finally, we demonstrate the effectiveness of self-training by directly generating pseudo-labels with an end-to-end model instead of a cascade model.

Motivation & Objective

• Address data scarcity in end-to-end speech translation by leveraging unlabeled audio data.
• Investigate the effectiveness of pseudo-labeling using cascade (ASR+MT) and end-to-end models for generating synthetic training labels.
• Demonstrate that self-training outperforms traditional encoder pre-training on the ASR task.
• Evaluate the impact of pseudo-label quality and model architecture scaling on performance.
• Simplify model training by replacing cascade systems with end-to-end models for pseudo-label generation.

Proposed method

• Generate pseudo-labels from unlabeled audio using a strong cascade ASR and MT system, then fine-tune an end-to-end ST model on the pseudo-labeled data.
• Train an end-to-end ST model on the same unlabeled data to generate pseudo-labels, enabling fully end-to-end self-training.
• Use a VggT and VggTLarge Transformer architecture for both the ST model and the pseudo-labeling model, with shared encoder and decoder components.
• Apply data filtering to remove very short or long utterances and use SentencePiece for subword tokenization with a 10,000-vocabulary unigram model.
• Fine-tune the final ST model on the original supervised data after pseudo-labeling to improve generalization.
• Use Adam optimizer with learning rate schedules specific to each architecture (LSTM: fixed LR; Transformer: original schedule).

Experimental results

Research questions

• RQ1 Can self-training with pseudo-labels from unlabeled audio significantly improve end-to-end speech translation performance in low- and high-resource settings?
• RQ2 How does the quality of pseudo-labels affect the performance of the final ST model?
• RQ3 Is self-training with pseudo-labels more effective than pre-training the encoder on the ASR task?
• RQ4 Can end-to-end ST models be used to generate pseudo-labels instead of relying on a cascade ASR+MT system?
• RQ5 Does scaling up model size with pseudo-labeled data lead to further performance gains?

Key findings

• Self-training with pseudo-labels from a cascade model yields 8.3 BLEU gain on MuST-C English-French and 5.7 BLEU gain on English-German, achieving state-of-the-art results.
• On the high-resource setting with 17,607 hours of LibriLight data, pseudo-labeling achieved 23.9 BLEU on En-De, outperforming encoder pre-training by 2.9 BLEU.
• Pseudo-labeling with a VggT end-to-end model for label generation outperformed the cascade model by 0.7 BLEU in the high-resource setting.
• The quality of pseudo-labels directly impacts performance: higher-quality ASR transcripts (7.3 WER) led to higher BLEU scores than lower-quality ones (27.7 WER).
• Self-training with only supervised data (pure self-training) still improved performance over the baseline, showing that end-to-end models can bootstrap the process.
• The method enabled training of larger models (e.g., VggTLarge) that were otherwise difficult to train, leading to significant performance gains.