[Paper Review] Sequence-to-Sequence Models Can Directly Translate Foreign Speech

Ron J. Weiss, Jan Chorowski+3 more|arXiv (Cornell University)|Mar 24, 2017

Natural Language Processing TechniquesComputer Science32 references54 citations

TL;DR

An end-to-end sequence-to-sequence model translates speech in one language directly to text in another without source-language transcripts, outperforming cascaded ASR+MT systems; multi-task training with ASR further boosts performance.

ABSTRACT

We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during training. We apply a slightly modified sequence-to-sequence with attention architecture that has previously been used for speech recognition and show that it can be repurposed for this more complex task, illustrating the power of attention-based models. A single model trained end-to-end obtains state-of-the-art performance on the Fisher Callhome Spanish-English speech translation task, outperforming a cascade of independently trained sequence-to-sequence speech recognition and machine translation models by 1.8 BLEU points on the Fisher test set. In addition, we find that making use of the training data in both languages by multi-task training sequence-to-sequence speech translation and recognition models with a shared encoder network can improve performance by a further 1.4 BLEU points.

Motivation & Objective

• Motivate end-to-end speech translation to avoid error accumulation in cascades (ASR+MT).
• Demonstrate that a single seq2seq with attention can translate audio in one language to text in another without source transcripts.
• Explore the impact of multi-task training by sharing encoders with ASR/MT tasks to improve translation quality.
• Show that end-to-end ST can achieve state-of-the-art BLEU on Fisher Callhome Spanish–English translation tasks.

Proposed method

• Use an attention-based seq2seq model with an encoder–decoder architecture and an attention mechanism to align input speech frames to output text tokens.
• Process log mel filterbank features with a convolutional and bidirectional LSTM encoder to produce h1..L representations.
• Decode with a multi-layer LSTM decoder guided by an attention-derived context vector c_k to produce output tokens y_k.
• Train end-to-end for speech translation and for ASR with a shared encoder in a multi-task setup to improve ST performance.
• Back-end training uses teacher forcing, adaptive optimization (Adam), gradient noise, and beam search decoding without external language models.

Experimental results

Research questions

• RQ1 Can an end-to-end seq2seq model translate foreign speech directly into target-language text without source-language transcripts during training?
• RQ2 Does sharing an encoder between speech translation and recognition tasks improve translation accuracy?
• RQ3 How does end-to-end ST compare to a cascade of ASR followed by MT on Fisher Callhome Spanish–English data?
• RQ4 What is the impact of decoder depth and multi-task training on ST performance?
• RQ5 What are the effects of training strategies (one-to-many vs many-to-one) on multi-task ST/ASR?

Key findings

• End-to-end speech translation outperforms the ASR→NMT cascade by about 1.8 BLEU points on Fisher/test.
• Multi-task ST/ASR with a shared encoder yields an additional ~1.4 BLEU point improvement across Fisher datasets.
• Increasing decoder depth up to four layers improves ST performance (BLEU) on Fisher/dev, with diminishing returns beyond that.
• Sharing all encoder layers in the multi-task setting yields the best ST performance, suggesting the encoder learns an interlingual representation.
• The end-to-end ST model achieves 46.5–47.3 BLEU on Fisher dev/test and 16.4–16.6 BLEU on Callhome devtest/evaltest, outperforming prior cascaded approaches on Fisher, while underperforming word-level MT baselines on Callhome due to data size.