[Paper Review] Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions

Awni Hannun, Ann Lee+2 more|arXiv (Cornell University)|Apr 4, 2019

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a fully convolutional sequence-to-sequence speech recognition model using time-depth separable (TDS) convolutions to achieve state-of-the-art WER on LibriSpeech while being over 10× more efficient than an RNN baseline. The TDS block decouples temporal and channel-wise aggregation, enabling large receptive fields with few parameters, and a stable beam search with a convolutional language model further improves accuracy, reducing WER by over 22% on noisy LibriSpeech compared to prior sequence-to-sequence models.

ABSTRACT

We propose a fully convolutional sequence-to-sequence encoder architecture with a simple and efficient decoder. Our model improves WER on LibriSpeech while being an order of magnitude more efficient than a strong RNN baseline. Key to our approach is a time-depth separable convolution block which dramatically reduces the number of parameters in the model while keeping the receptive field large. We also give a stable and efficient beam search inference procedure which allows us to effectively integrate a language model. Coupled with a convolutional language model, our time-depth separable convolution architecture improves by more than 22% relative WER over the best previously reported sequence-to-sequence results on the noisy LibriSpeech test set.

Motivation & Objective

• To develop a highly efficient and accurate sequence-to-sequence speech recognition model that outperforms strong RNN-based baselines.
• To reduce model complexity and inference time by replacing recurrent layers with a fully convolutional encoder using time-depth separable convolutions.
• To enable stable and scalable beam search inference that effectively integrates a convolutional language model without accuracy degradation at large beam sizes.
• To eliminate inefficient sequential components like scheduled sampling and location-based attention while maintaining or improving recognition performance.

Proposed method

• The encoder uses a time-depth separable (TDS) convolution block that first applies a 2D convolution over time with kernel size k×1, followed by a fully connected block with two 1×1 convolutions and ReLU activations.
• The TDS block includes residual connections and layer normalization applied over all dimensions, including time, to stabilize training and improve generalization.
• The model employs three sub-sampling layers with stride 2, increasing channel count to compress temporal information while maintaining context.
• The decoder uses teacher forcing during training and a simplified inner-product attention mechanism that enables parallel computation, replacing costly neural attention.
• A stable beam search procedure is developed that maintains accuracy even at beam sizes up to 80, allowing effective integration with a convolutional language model.
• The convolutional language model is integrated during inference to improve n-gram baseline performance, with beam search efficiently handling the language model constraints.

Experimental results

Research questions

• RQ1 Can a fully convolutional encoder with time-depth separable convolutions achieve better WER than RNN-based sequence-to-sequence models while being significantly more efficient?
• RQ2 Does removing sequential dependencies like scheduled sampling and location-based attention degrade performance, or can simpler alternatives maintain accuracy?
• RQ3 Can a stable beam search procedure be designed to scale effectively with large beam sizes when integrating a convolutional language model?
• RQ4 How does the size of the receptive field impact WER, and what is the minimum threshold for reliable sequence-to-sequence recognition?
• RQ5 To what extent can a convolutional language model improve WER when combined with an efficient beam search and a TDS-based encoder?

Key findings

• The TDS convolution model achieves a WER of 5.10% on LibriSpeech dev-clean and 14.99% on dev-other, outperforming the best previously reported sequence-to-sequence results.
• The model reduces WER by over 22% relative on the noisy LibriSpeech test set compared to prior sequence-to-sequence models, achieving state-of-the-art performance.
• Training one epoch on LibriSpeech takes only 7 minutes, making the model more than 10× faster than an RNN baseline with the same sub-sampling factor.
• Beam search with a beam size of 80 maintains high accuracy and shows consistent WER improvements, especially on the dev-other set, when using a convolutional language model.
• The model's WER degrades sharply when the receptive field drops below a threshold, indicating that sufficient context is critical for disambiguating decoder queries.
• The absence of scheduled sampling and location-based attention does not harm performance, and the model achieves strong results using simpler, more efficient alternatives.