[Paper Review] End-to-End Automatic Speech Translation of Audiobooks

Alexandre Bérard, Laurent Besacier+2 more|arXiv (Cornell University)|Feb 12, 2018

Natural Language Processing Techniques20 references126 citations

TL;DR

This paper presents end-to-end automatic speech translation (AST) models for English-to-French audiobooks using an augmented LibriSpeech corpus with 236 hours of aligned English speech and French translations. It compares cascaded ASR+MT systems with end-to-end models that either bypass source transcription entirely or use it during training, showing that pre-training and multi-task learning improve performance, though cascaded models still outperform end-to-end ones on real speech.

ABSTRACT

We investigate end-to-end speech-to-text translation on a corpus of audiobooks specifically augmented for this task. Previous works investigated the extreme case where source language transcription is not available during learning nor decoding, but we also study a midway case where source language transcription is available at training time only. In this case, a single model is trained to decode source speech into target text in a single pass. Experimental results show that it is possible to train compact and efficient end-to-end speech translation models in this setup. We also distribute the corpus and hope that our speech translation baseline on this corpus will be challenged in the future.

Motivation & Objective

• Investigate end-to-end speech-to-text translation on a large-scale, real-world audiobook corpus with parallel speech and target text.
• Compare the performance of end-to-end AST models that do not use source language transcription during training or inference with those that use it during training only.
• Establish a strong baseline for future research on end-to-end AST using a newly created, publicly available audiobook corpus.
• Evaluate the effectiveness of pre-training and multi-task learning in improving compact end-to-end AST models.
• Demonstrate that even with limited source transcription access, end-to-end models can achieve competitive performance on real speech.

Proposed method

• Augment the LibriSpeech corpus by aligning English audiobook speech with French e-book translations, creating 236 hours of parallel speech and target text.
• Use a hybrid speech encoder combining convolutional layers and bidirectional LSTMs to extract hierarchical features from raw audio (MFCCs).
• Implement a character-level decoder with a conditional LSTM and global attention mechanism to generate target text token-by-token.
• Train end-to-end AST models using a single encoder-decoder architecture, sharing the encoder with ASR and the decoder with MT for multi-task learning.
• Apply multi-task training by alternating updates between AST, ASR, and MT tasks, with 60% updates for AST and 20% each for ASR and MT.
• Use variational dropout and input truncation (1400 frames for speech, 300 characters for text) to manage training stability and memory constraints.

Experimental results

Research questions

• RQ1 Can end-to-end speech-to-text translation models achieve competitive performance on real audiobook speech without requiring source language transcription during inference?
• RQ2 How does the availability of source language transcription during training affect the performance of end-to-end AST models compared to the fully end-to-end setting?
• RQ3 To what extent do pre-training and multi-task learning improve the performance of compact end-to-end AST models on large-scale real speech?
• RQ4 How does the performance of end-to-end AST compare to cascaded ASR+MT pipelines on real audiobook speech?
• RQ5 Is the proposed augmented LibriSpeech corpus a viable and challenging benchmark for future end-to-end AST research?

Key findings

• On the BTEC synthetic corpus, the cascaded ASR+MT system achieved a BLEU score of 43.8, outperforming the best end-to-end model (37.9 BLEU) and the pre-trained model (40.4 BLEU).
• On the real audiobook corpus (Augmented LibriSpeech), the cascaded model achieved 15.8 BLEU, while the best end-to-end model reached 15.5 BLEU, showing strong performance despite the challenging real-world data.
• Pre-training significantly accelerated convergence, with the pre-trained model reaching a 13.3 BLEU score on LibriSpeech after only 129k steps, compared to 369k steps for the end-to-end model.
• Multi-task training did not improve performance when combined with pre-training, suggesting that pre-training alone is sufficient for faster convergence in this setup.
• The use of a character-level decoder improved results compared to prior work, with a 10.4M parameter end-to-end model achieving 31.3 BLEU on BTEC, compared to 29.1 BLEU in the prior study.
• The corpus is challenging: even Google Translate scored only 22.2 BLEU on the LibriSpeech test set, indicating that high-quality translation on real audiobook speech remains difficult.