[Paper Review] Thank you for Attention: A survey on Attention-based Artificial Neural Networks for Automatic Speech Recognition

Priyabrata Karmakar, Shyh Wei Teng+1 more|arXiv (Cornell University)|Feb 14, 2021

Speech Recognition and Synthesis76 references11 citations

TL;DR

This survey provides a comprehensive analysis of attention-based neural networks in automatic speech recognition (ASR), covering both RNN- and Transformer-based architectures for offline and streaming ASR. It details the evolution of global, local, content-based, location-based, and hybrid attention mechanisms, and explores streaming-friendly adaptations like chunk-flow, time-restricted self-attention, and monotonic attention to enable real-time inference while maintaining performance.

ABSTRACT

Attention is a very popular and effective mechanism in artificial neural network-based sequence-to-sequence models. In this survey paper, a comprehensive review of the different attention models used in developing automatic speech recognition systems is provided. The paper focuses on the development and evolution of attention models for offline and streaming speech recognition within recurrent neural network- and Transformer- based architectures.

Motivation & Objective

• To provide a systematic review of attention mechanisms in end-to-end automatic speech recognition (ASR) systems.
• To analyze the evolution of attention models in both recurrent neural network (RNN)-based and Transformer-based ASR architectures.
• To examine techniques enabling streaming ASR using attention mechanisms, including chunking and restricted attention spans.
• To identify challenges in online ASR, such as alignment learning inefficiency and attention head redundancy, and review proposed solutions.
• To bridge the gap in existing literature by offering a focused, comprehensive survey on attention in ASR, distinct from broader NLP surveys.

Proposed method

• Categorize attention mechanisms into global/soft, local/hard, content-based, location-based, and hybrid types based on attention computation and scope.
• Explain the role of self-attention in Transformer-based ASR, replacing recurrence and enabling parallel sequence processing.
• Detail streaming adaptations such as chunk-flow mechanism (Eq. 21), which restricts self-attention to fixed-length input windows (e.g., Nl=20, Nr=10).
• Describe time-restricted self-attention that limits attention to a fixed number of frames on left and right (e.g., 15 and 6 frames), enabling real-time inference.
• Present synchronous Transformer and monotonic truncated attention, where attention is restricted to left-context only and processed chunkwise.
• Introduce HeadDrop regularization and pruning techniques to improve alignment learning in monotonic multihead attention (MMA) by masking redundant heads and forcing effective learning.

Experimental results

Research questions

• RQ1 How have different types of attention mechanisms (global, local, content-based, location-based, hybrid) evolved and been applied in RNN- and Transformer-based ASR models?
• RQ2 What are the key architectural and training modifications required to adapt attention mechanisms for streaming (online) speech recognition?
• RQ3 How do techniques like chunk-flow, time-restricted attention, and monotonic attention enable real-time ASR while preserving performance?
• RQ4 What challenges arise in online ASR with multi-head attention, and how do methods like HeadDrop and pruning address them?
• RQ5 How do self-attention mechanisms in Transformers overcome the limitations of recurrence in sequence modeling for ASR?

Key findings

• The chunk-flow mechanism restricts self-attention to a fixed window (e.g., 20 frames left, 10 right), enabling streaming inference with acceptable performance despite limited context.
• Time-restricted self-attention limits attention to 15 frames on the left and 6 on the right, achieving real-time inference with minimal performance degradation.
• Monotonic multihead attention (MMA) enables online decoding by restricting each head to attend only left-to-right positions, but only a subset of heads (dominant ones) effectively learn alignments.
• HeadDrop regularization improves alignment learning by randomly masking a portion of attention heads, forcing the remaining heads to learn more effectively and reducing redundancy.
• Pruning redundant MA heads in lower layers enhances coordination among attention heads and improves inference efficiency in online ASR.
• Synchronous Transformer and chunk-based self-attention allow streaming by processing encoded frames in overlapped chunks and restricting attention to left-context only, avoiding full-sequence dependency.