[Paper Review] Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation

Alexandre Bérard, Olivier Pietquin+2 more|arXiv (Cornell University)|Dec 6, 2016

Natural Language Processing Techniques13 references208 citations

TL;DR

The paper presents an end-to-end speech-to-text translation system based on attention-based encoder-decoder networks, comparing speech translation to text translation and evaluating on a small synthetic French–English corpus.

ABSTRACT

This paper proposes a first attempt to build an end-to-end speech-to-text translation system, which does not use source language transcription during learning or decoding. We propose a model for direct speech-to-text translation, which gives promising results on a small French-English synthetic corpus. Relaxing the need for source language transcription would drastically change the data collection methodology in speech translation, especially in under-resourced scenarios. For instance, in the former project DARPA TRANSTAC (speech translation from spoken Arabic dialects), a large effort was devoted to the collection of speech transcripts (and a prerequisite to obtain transcripts was often a detailed transcription guide for languages with little standardized spelling). Now, if end-to-end approaches for speech-to-text translation are successful, one might consider collecting data by asking bilingual speakers to directly utter speech in the source language from target language text utterances. Such an approach has the advantage to be applicable to any unwritten (source) language.

Motivation & Objective

• Motivate the study of end-to-end speech-to-text translation that does not rely on source-language transcripts.
• Propose and compare two end-to-end models for text translation and for speech translation using attention mechanisms.
• Assess whether training on a small, specialized corpus is feasible for end-to-end translation.
• Demonstrate the potential robustness to inter-speaker variability using synthetic speech data.

Proposed method

• Use attention-based encoder-decoder neural networks for both text translation and speech translation.
• Employ a bidirectional LSTM encoder and a two-layer LSTM decoder with attention to generate target sequences.
• For text input, apply a Bahdanau-style attention mechanism; for speech input, use a convolutional attention model with memory of previous attention via a convolution filter.
• Train with Adam optimization and apply dropout between encoder and decoder layers.
• Implement a hierarchical encoder for the speech model to reduce input sequence length, and use a 40 MFCC feature representation for speech inputs.
• Evaluate with greedy decoding and beam-search decoding, and compare against a traditional SMT baseline.

Experimental results

Research questions

• RQ1 Can an end-to-end speech-to-text translation model be trained without relying on source-language transcripts?
• RQ2 How does end-to-end speech translation performance compare to text translation and to a pipeline SMT baseline on a small synthetic French–English corpus?
• RQ3 Does the end-to-end approach generalize to a new speaker without explicit speaker adaptation?
• RQ4 What is the impact of decoding strategies (greedy vs beam search with/without language model) on translation quality?

Key findings

• End-to-end speech translation on a small synthetic French–English corpus yields BLEU scores that are competitive with a baseline SMT system under several decoding settings.
• An ensemble of five models with a language model achieves BLEU scores approaching the SMT baseline on the dev and test sets.
• The speech translation model remains reasonably robust to a new speaker not included in training, indicating potential generalization without speaker adaptation.
• Training time is short (about 2 hours for text models and 8 hours for speech models on a GTX 1070), demonstrating feasibility for rapid experimentation.
• The study confirms that end-to-end models can learn alignment and translation jointly, as visualized by attention alignment figures.