[Paper Review] Transformers with convolutional context for ASR

Abdelrahman Mohamed, Dmytro Okhonko+1 more|arXiv (Cornell University)|Apr 26, 2019

Natural Language Processing Techniques25 references152 citations

TL;DR

The paper replaces sinusoidal positional embeddings in transformers for ASR with convolutionally learned input representations, enabling stable training and improved WER on Librispeech without external language models.

ABSTRACT

The recent success of transformer networks for neural machine translation and other NLP tasks has led to a surge in research work trying to apply it for speech recognition. Recent efforts studied key research questions around ways of combining positional embedding with speech features, and stability of optimization for large scale learning of transformer networks. In this paper, we propose replacing the sinusoidal positional embedding for transformers with convolutionally learned input representations. These contextual representations provide subsequent transformer blocks with relative positional information needed for discovering long-range relationships between local concepts. The proposed system has favorable optimization characteristics where our reported results are produced with fixed learning rate of 1.0 and no warmup steps. The proposed model achieves a competitive 4.7% and 12.9% WER on the Librispeech ``test clean'' and ``test other'' subsets when no extra LM text is provided.

Motivation & Objective

• Investigate how to introduce positional information for speech features in transformer-based ASR.
• Propose convolutional context as input processing before transformer blocks to provide relative positional signals.
• Examine optimization stability and performance of a deep encoder with convolutional context in ASR.
• Evaluate on Librispeech without external language model data to assess orthogonality to LM improvements.

Proposed method

• Replace sinusoidal positional embeddings with 2-D convNet encoders and 1-D convNet decoders to learn contextual input representations.
• Use two 2-D convolutional blocks in the encoder and multiple encoder transformer blocks with deep architecture.
• Apply 1-D convolutional blocks on decoder embeddings with separate decoder transformer blocks and attention layers.
• Avoid learning rate warmup; use AdaDelta with fixed learning rate of 1.0 and gradient clipping.
• Train end-to-end on Librispeech 1000h with 5k unigram subword units; report averaged last 30 checkpoints over 80 epochs.

Experimental results

Research questions

• RQ1 Does convolutional context provide the necessary relative positional information for transformers to model long-range dependencies in speech?
• RQ2 How does decoder convolutional context depth and encoder/decoder depth affect WER on Librispeech scenarios (dev/test, clean/other)?
• RQ3 What is the impact of removing sinusoidal positional embeddings in favor of learned convolutional context on ASR performance?
• RQ4 Can a deep encoder with convolutional context achieve competitive WER without external language models?

Key findings

• The proposed convolutional context improves WER over sinusoidal position embeddings in several configurations.
• Best configurations with deep encoders (e.g., 16 encoder blocks) and multi-layer decoder attention achieve 12% and 16% relative WER reductions on dev other and test other without extra LM data.
• Decoder context depth and broader encoder depth yield better WER than shallow setups; broad context across multiple conv layers is advantageous.
• The model achieves 4.7 and 12.9 WER on Librispeech test clean and test other when no external LM text is used, compared to prior results.
• Using a fixed learning rate of 1.0 with AdaDelta and no warmup still yields stable convergence and strong performance.
• Combining this approach with future LM-based improvements (e.g., OCD) could yield further gains.