[Paper Review] Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning

Suyoun Kim, Takaaki Hori+1 more|arXiv (Cornell University)|Sep 21, 2016

Speech Recognition and Synthesis12 references18 citations

TL;DR

This paper proposes a multi-task learning framework that jointly trains a CTC and attention-based end-to-end ASR model using a shared encoder, improving alignment robustness and convergence speed. By combining CTC's left-to-right monotonic constraints with attention's context-aware decoding, the method achieves 5.4–14.6% relative CER improvements over CTC and attention baselines on WSJ and CHiME-4, with faster learning and better generalization in noisy and clean conditions.

ABSTRACT

Recently, there has been an increasing interest in end-to-end speech recognition that directly transcribes speech to text without any predefined alignments. One approach is the attention-based encoder-decoder framework that learns a mapping between variable-length input and output sequences in one step using a purely data-driven method. The attention model has often been shown to improve the performance over another end-to-end approach, the Connectionist Temporal Classification (CTC), mainly because it explicitly uses the history of the target character without any conditional independence assumptions. However, we observed that the performance of the attention has shown poor results in noisy condition and is hard to learn in the initial training stage with long input sequences. This is because the attention model is too flexible to predict proper alignments in such cases due to the lack of left-to-right constraints as used in CTC. This paper presents a novel method for end-to-end speech recognition to improve robustness and achieve fast convergence by using a joint CTC-attention model within the multi-task learning framework, thereby mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks demonstrates its advantages over both the CTC and attention-based encoder-decoder baselines, showing 5.4-14.6% relative improvements in Character Error Rate (CER).

Motivation & Objective

• Address the instability and slow convergence of attention-based end-to-end ASR in noisy environments and long sequences.
• Overcome the lack of left-to-right alignment constraints in pure attention models, which leads to misalignment and poor generalization.
• Improve training stability and convergence speed by leveraging CTC's monotonic alignment in a multi-task learning framework.
• Enhance performance on both clean (WSJ) and noisy (CHiME-4) speech recognition tasks without external language models.
• Demonstrate that CTC's auxiliary objective improves generalization even in clean data scenarios.

Proposed method

• Train a shared encoder with both CTC and attention-based decoder objectives simultaneously using multi-task learning.
• Use a weighted combination of CTC loss and attention loss, with hyperparameter λ controlling the balance between the two objectives.
• Apply the CTC loss via the forward-backward algorithm to enforce monotonic, left-to-right alignments in the encoder output.
• Use a location-based attention mechanism in the decoder to attend over encoder states, conditioned on previous decoder outputs.
• Implement the model using a 4-layer bidirectional LSTM encoder and a 1-layer LSTM decoder, with linear projections and residual connections between layers.
• Optimize using AdaDelta with gradient clipping and uniform weight initialization, and decode using beam search with length normalization.

Experimental results

Research questions

• RQ1 Can joint training of CTC and attention improve alignment robustness in noisy speech recognition?
• RQ2 Does incorporating CTC's monotonic alignment constraint accelerate convergence in end-to-end ASR?
• RQ3 How does the multi-task learning framework perform compared to standalone CTC and attention models on both clean and noisy datasets?
• RQ4 What is the optimal balance between CTC and attention losses for maximizing ASR performance?
• RQ5 Does the CTC objective improve generalization in clean data settings despite not modeling character dependencies?

Key findings

• The joint CTC-attention model achieved 5.4–14.6% relative CER improvements over both CTC and attention baselines on WSJ and CHiME-4 datasets.
• The model with λ = 0.2 achieved the best performance on both WSJ and CHiME-4, indicating optimal balance between CTC and attention objectives.
• Learning curves showed that the MTL model reached higher character accuracy faster than the pure attention model, with convergence occurring earlier.
• Visualization of attention alignments revealed that the MTL model learned correct monotonic alignments by the 5th epoch, while the attention model failed to converge even by the 9th epoch.
• The CTC loss effectively guided the attention mechanism toward monotonic alignments, reducing misalignment in early training.
• Even on clean WSJ data, the MTL model outperformed both baselines, suggesting CTC’s training inductive bias improves generalization.