[Paper Review] A Transformer with Interleaved Self-attention and Convolution for Hybrid Acoustic Models

Liang Lu|arXiv (Cornell University)|Oct 23, 2019

Speech Recognition and Synthesis25 references4 citations

TL;DR

This paper proposes a hybrid acoustic model that interleaves self-attention and 1D convolutional layers in a Transformer architecture, improving convergence speed and recognition accuracy on Librispeech. The model achieves competitive results—comparable to Kaldi's TDNN baseline—without speaker adaptation or data augmentation, demonstrating the effectiveness of architectural interleaving for speech recognition.

ABSTRACT

Transformer with self-attention has achieved great success in the area of nature language processing. Recently, there have been a few studies on transformer for end-to-end speech recognition, while its application for hybrid acoustic model is still very limited. In this paper, we revisit the transformer-based hybrid acoustic model, and propose a model structure with interleaved self-attention and 1D convolution, which is proven to have faster convergence and higher recognition accuracy. We also study several aspects of the transformer model, including the impact of the positional encoding feature, dropout regularization, as well as training with and without time restriction. We show competitive recognition results on the public Librispeech dataset when compared to the Kaldi baseline at both cross entropy training and sequence training stages. For reproducible research, we release our source code and recipe within the PyKaldi2 toolbox.

Motivation & Objective

• To explore the application of Transformer-based models in hybrid acoustic modeling, a framework less studied than end-to-end approaches.
• To address the challenge of training stability and long-sequence modeling in hybrid models by integrating convolutional layers with self-attention.
• To investigate the impact of positional encoding, dropout, and time-restriction on model performance in a hybrid setting.
• To demonstrate that interleaved attention and convolution improve convergence and recognition accuracy compared to pure self-attention models.
• To provide a reproducible implementation via PyKaldi2, supporting future research in hybrid Transformer models.

Proposed method

• Design a Transformer architecture where self-attention and 1D convolutional layers are alternated in the encoder stack to balance local feature learning and long-range dependency modeling.
• Apply layer normalization after each sub-layer (multi-head self-attention and feed-forward network) and use residual connections to stabilize training.
• Use sinusoidal positional encoding to inject sequential order information into the input embeddings, enabling the model to learn relative position dependencies.
• Train the model using both cross-entropy (CE) and sequence training with the maximum mutual information (MMI) criterion to improve alignment and generalization.
• Apply time-restriction in self-attention by limiting the context window to a fixed number of frames, reducing computational cost and improving training stability.
• Use a 6-layer encoder with multi-head attention (8 heads) and feed-forward networks with d_model=256, d_ff=1024, and dropout rate of 0.1.

Experimental results

Research questions

• RQ1 How does interleaving self-attention and convolution in a Transformer encoder affect convergence speed and recognition accuracy in hybrid acoustic models?
• RQ2 What is the impact of positional encoding and dropout regularization on the performance of self-attention-based hybrid models?
• RQ3 Does training with time-restricted self-attention improve model stability and recognition performance compared to full-sequence attention?
• RQ4 Can a pure Transformer-based hybrid model achieve performance competitive with strong Kaldi-based TDNN baselines without speaker-level adaptation or data augmentation?
• RQ5 How does the model perform under sequence training with the MMI criterion, and does it generalize well to noisy test sets?

Key findings

• The interleaved self-attention and convolution model achieved faster convergence and higher recognition accuracy than a pure self-attention model.
• On the Librispeech test-other set, the CE-trained Transformer model achieved a WER of 11.8%, comparable to the Kaldi TDNN baseline (12.1% for CE and 11.3% for sMBR).
• After sequence training with MMI, the Transformer model reduced WER to 11.1% on test-other, outperforming the Kaldi TDNN system on the same split (11.3% for sMBR).
• Increasing the future context in self-attention beyond a limited window did not improve WER despite reducing CE loss, suggesting potential overfitting or attention head misalignment.
• The model with time-restricted attention showed slightly worse performance than the full-attention version, but the difference was small, indicating that local context is sufficient for good performance.
• The model achieved competitive results without using i-vector adaptation or speed perturbation, highlighting the robustness of the proposed architecture.