[Paper Review] On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformers

Shucong Zhang, Erfan Loweimi+2 more|arXiv (Cornell University)|Nov 8, 2020

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper investigates whether self-attention in upper Transformer encoder layers is necessary for automatic speech recognition. By replacing upper self-attention layers with feed-forward networks and introducing a diagonality metric, the authors find that upper layers naturally attend locally, making self-attention redundant there—resulting in no performance drop and minor gains when replaced.

ABSTRACT

Self-attention models such as Transformers, which can capture temporal relationships without being limited by the distance between events, have given competitive speech recognition results. However, we note the range of the learned context increases from the lower to upper self-attention layers, whilst acoustic events often happen within short time spans in a left-to-right order. This leads to a question: for speech recognition, is a global view of the entire sequence useful for the upper self-attention encoder layers in Transformers? To investigate this, we train models with lower self-attention/upper feed-forward layers encoders on Wall Street Journal and Switchboard. Compared to baseline Transformers, no performance drop but minor gains are observed. We further developed a novel metric of the diagonality of attention matrices and found the learned diagonality indeed increases from the lower to upper encoder self-attention layers. We conclude the global view is unnecessary in training upper encoder layers.

Motivation & Objective

• To determine whether global self-attention in upper Transformer encoder layers is essential for automatic speech recognition.
• To investigate the discrepancy between the success of global self-attention in Transformers and the performance gains from monotonic (local) attention in RNN-based models.
• To analyze how the context range learned by self-attention layers evolves from lower to upper layers in the encoder.
• To develop a novel metric for measuring attention matrix diagonality to quantify local vs. global attention behavior.
• To explore architectural efficiency by replacing upper self-attention layers with feed-forward layers without performance degradation.

Proposed method

• Trained Transformer models with upper feed-forward layers and lower self-attention layers on WSJ and Switchboard datasets.
• Proposed a novel diagonality metric to quantify how much attention heads focus on local, left-to-right time spans in the input sequence.
• Computed average diagonality across all attention heads per encoder layer to track the evolution of attention patterns.
• Compared model performance after replacing varying numbers of upper self-attention layers with feed-forward layers.
• Analyzed attention matrix diagonality trends before and after replacing layers to assess structural stability and attention behavior.

Experimental results

Research questions

• RQ1 Is the global context modeling capability of self-attention necessary in upper Transformer encoder layers for speech recognition?
• RQ2 How does the effective receptive field of self-attention layers change from lower to upper layers in a Transformer encoder?
• RQ3 Can replacing upper self-attention layers with feed-forward layers maintain or improve ASR performance?
• RQ4 To what extent do self-attention heads in upper layers learn to focus on local, left-to-right contexts despite having global access?
• RQ5 Can diagonality of attention matrices serve as a reliable indicator for identifying redundant self-attention layers?

Key findings

• Replacing upper self-attention layers with feed-forward layers results in no performance degradation and minor improvements on WSJ and Switchboard.
• The average diagonality of attention matrices increases from lower to upper encoder layers, indicating a shift toward local, left-to-right attention in upper layers.
• The 9th and 10th layers in a 12-layer encoder still require self-attention, as replacing them with feed-forward layers increases error rates.
• The topmost two layers (11th and 12th) exhibit near-maximal diagonality (~1), showing they already attend only to local contexts, making self-attention redundant there.
• The 7th and 8th layers also show high diagonality, suggesting they may be candidates for replacement, though replacing them with feed-forward layers increases error rates, likely due to disruption of context flow.
• After replacing upper layers with feed-forward networks, the overall trend of increasing diagonality from lower to upper layers is preserved, confirming stable attention behavior.