[Paper Review] Self-Supervised Representations Improve End-to-End Speech Translation

Anne Wu, Changhan Wang+2 more|arXiv (Cornell University)|Jun 22, 2020

Natural Language Processing Techniques31 references4 citations

TL;DR

This paper demonstrates that self-supervised speech representations—specifically wav2vec, vq-wav2vec, and BERT features—significantly improve end-to-end speech translation (ST) performance in both high- and low-resource settings. Pre-trained on English unlabeled data, these features transfer effectively to other languages with minimal tuning, and combining them with ASR pre-training further boosts performance, especially in low-resource scenarios.

ABSTRACT

End-to-end speech-to-text translation can provide a simpler and smaller system but is facing the challenge of data scarcity. Pre-training methods can leverage unlabeled data and have been shown to be effective on data-scarce settings. In this work, we explore whether self-supervised pre-trained speech representations can benefit the speech translation task in both high- and low-resource settings, whether they can transfer well to other languages, and whether they can be effectively combined with other common methods that help improve low-resource end-to-end speech translation such as using a pre-trained high-resource speech recognition system. We demonstrate that self-supervised pre-trained features can consistently improve the translation performance, and cross-lingual transfer allows to extend to a variety of languages without or with little tuning.

Motivation & Objective

• To investigate whether self-supervised pre-trained speech representations improve end-to-end speech translation in both high- and low-resource settings.
• To evaluate the cross-lingual transferability of these representations to low-resource languages without or with minimal fine-tuning.
• To examine the effectiveness of combining self-supervised features with pre-trained ASR systems for low-resource ST.
• To assess whether ASR performance is a reliable proxy for ST performance when using pre-training methods.

Proposed method

• Utilized self-supervised pre-training methods: wav2vec (contrastive predictive coding), vq-wav2vec (vector-quantized representation), and BERT (on discretized representations).
• Applied these pre-trained features as input representations in a sequence-to-sequence BiLSTM encoder-decoder model with attention for ST.
• Explored a hybrid BERT-backbone architecture where pre-trained BERT models were used as encoders for low-resource ST tasks.
• Fine-tuned pre-trained features on Common Voice data for target languages (e.g., French, Chinese) to improve cross-lingual transfer.
• Combined pre-trained ASR models (trained on English and target language data) to warm-start the ST encoder, improving generalization in low-resource settings.
• Used log-mel filterbanks as baseline features for comparison across all experiments.

Experimental results

Research questions

• RQ1 Can self-supervised speech representations improve end-to-end speech translation performance in high-resource and low-resource settings?
• RQ2 How well do self-supervised features transfer to low-resource languages without additional tuning?
• RQ3 Does combining self-supervised features with ASR pre-training yield better ST performance than using either method alone?
• RQ4 Is ASR performance on a pre-training task a reliable indicator of downstream ST performance?

Key findings

• Self-supervised features (wav2vec, vq-wav2vec, BERT) consistently improved ST performance over log-mel filterbanks in both English-to-X and X-to-English settings.
• Cross-lingual transfer of pre-trained English features to 11 low-resource languages yielded gains without fine-tuning, though performance gains were influenced by data quantity and linguistic similarity.
• Fine-tuning BERT features on target language Common Voice data improved BLEU scores by up to +1.3 on Fr-En and +1.1 on Zh-En ST compared to frozen features.
• Pre-training the ST encoder with a multilingual ASR model (English + target language) led to better ST performance than pre-training both encoder and decoder with English-only ASR, with gains of up to +1.6 BLEU on Es-En ST.
• In some cases, ASR models with higher WER (e.g., vq-wav2vec) still produced better ST results than models with lower WER (e.g., BERT), indicating that ASR WER is not a reliable predictor of ST performance.
• The BERT-backbone architecture achieved better performance than training from scratch on low-resource settings (e.g., 6.8 BLEU on Zh-En ST), though it was less effective than ASR pre-training.