[Paper Review] Attention-Based Models for Speech Recognition

Jan Chorowski, Dzmitry Bahdanau+3 more|arXiv (Cornell University)|Jun 24, 2015

Multimodal Machine Learning Applications31 references1,780 citations

TL;DR

This paper proposes an attention-based recurrent sequence generator for end-to-end speech recognition that enhances attention with location-aware convolutional features and a normalized alignment mechanism. The model achieves 17.6% phoneme error rate on TIMIT and generalizes robustly to utterances up to 11× longer than training data, outperforming baseline models that fail on long sequences due to location-tracking saturation.

ABSTRACT

Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks in- cluding machine translation, handwriting synthesis and image caption gen- eration. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the at- tention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.

Motivation & Objective

• To develop an end-to-end trainable speech recognition model using attention-based sequence generation.
• To address the failure of standard attention mechanisms on long speech inputs due to implicit location tracking.
• To improve robustness to long and noisy input sequences by introducing location-aware attention and alignment smoothing.
• To reduce over-concentration on single frames through a normalized attention mechanism.
• To evaluate the model on the TIMIT phoneme recognition benchmark with both standard and artificially extended utterances.

Proposed method

• Introduces a hybrid attention mechanism that combines content-based attention with location-aware features derived from convolving previous attention weights with trainable filters.
• Applies 1D convolutional filters to previous alignment vectors to generate auxiliary features that encode positional context.
• Implements a normalized attention mechanism using a softplus-activated dot-product to prevent over-concentration on single frames.
• Uses alignment sharpening techniques such as windowing and temperature scaling to improve forced alignment quality during decoding.
• Trains the model end-to-end using backpropagation through time with a cross-entropy loss on phoneme sequences.
• Evaluates performance on TIMIT with both original and concatenated (longer) utterances to test generalization.

Experimental results

Research questions

• RQ1 Can an attention-based sequence generator be effectively applied to speech recognition with long input sequences?
• RQ2 Why does a standard attention mechanism fail on long speech inputs despite good performance on short ones?
• RQ3 How can attention mechanisms be made robust to long sequences by incorporating positional awareness?
• RQ4 Does normalizing attention to prevent over-concentration on single frames improve recognition accuracy?
• RQ5 Can the proposed attention mechanism generalize to sequences much longer than those seen during training?

Key findings

• The baseline model achieved 18.7% phoneme error rate (PER) on standard TIMIT test sets but failed on longer, concatenated utterances due to implicit location tracking.
• The proposed location-aware attention mechanism reduced PER to 18.0% on single utterances and maintained below 20% PER on 11× longer utterances.
• Introducing a normalized attention mechanism further reduced PER to 17.6% by preventing over-concentration on individual frames.
• The model with convolutional location features successfully aligned sequences up to 200 phones long, while the baseline model failed after ~40 phones.
• Alignment sharpening techniques like windowing and temperature scaling significantly improved alignment quality, especially for the location-aware model.
• The location-aware model showed better robustness on repeated utterances than on concatenated ones, indicating sensitivity to irrelevant frames in diverse inputs.