[Paper Review] Very Deep Convolutional Networks for End-to-End Speech Recognition

Yu Zhang, William Chan+1 more|arXiv (Cornell University)|Oct 10, 2016

Speech Recognition and Synthesis19 references10 citations

TL;DR

This paper proposes a very deep convolutional neural network (CNN) with residual connections, batch normalization, network-in-network (NiN) modules, and convolutional LSTMs to significantly improve end-to-end speech recognition. By stacking 15 layers of deep CNNs and residual blocks before a bidirectional LSTM, the model achieves a 10.53% word error rate on the WSJ test set without language models or lexicons, representing a 24.7% relative improvement over the baseline.

ABSTRACT

Sequence-to-sequence models have shown success in end-to-end speech recognition. However these models have only used shallow acoustic encoder networks. In our work, we successively train very deep convolutional networks to add more expressive power and better generalization for end-to-end ASR models. We apply network-in-network principles, batch normalization, residual connections and convolutional LSTMs to build very deep recurrent and convolutional structures. Our models exploit the spectral structure in the feature space and add computational depth without overfitting issues. We experiment with the WSJ ASR task and achieve 10.5\% word error rate without any dictionary or language using a 15 layer deep network.

Motivation & Objective

• To improve end-to-end automatic speech recognition (ASR) by increasing the depth and expressiveness of the acoustic encoder beyond shallow RNNs.
• To apply deep learning techniques from computer vision—such as residual connections, batch normalization, and NiN—to speech recognition for better generalization and optimization.
• To explore the effectiveness of convolutional LSTMs in preserving spectral structure and reducing overfitting in sequence-to-sequence models.
• To achieve state-of-the-art performance on the WSJ ASR benchmark without language models or lexicons.

Proposed method

• The model uses a sequence-to-sequence architecture with an attention-based decoder and a deep CNN-based encoder.
• The encoder employs strided convolutions in early layers to reduce temporal dimensionality and GPU memory usage.
• Network-in-Network (NiN) modules are used with 1×1 convolutions to increase depth and non-linearities while reducing parameters.
• Batch normalization is applied to stabilize training and act as a regularizer, especially critical for deep models with high gradient variance.
• Residual connections are used to enable training of very deep networks (up to 15 layers) without degradation or poor optimization.
• Convolutional LSTMs replace standard LSTMs by using convolutions in the cell state, preserving spectral structure and reducing overfitting.

Experimental results

Research questions

• RQ1 Can very deep convolutional networks with residual connections and batch normalization improve end-to-end speech recognition performance?
• RQ2 How do NiN modules and 1×1 convolutions contribute to deeper, more parameter-efficient models in ASR?
• RQ3 Does replacing standard LSTMs with convolutional LSTMs improve modeling of spectral-temporal features in speech?
• RQ4 What is the impact of architectural depth and component integration (e.g., residual blocks, batch norm) on WER in end-to-end ASR?

Key findings

• The proposed model achieves a 10.53% word error rate on the WSJ eval92 set without language models or lexicons, a significant improvement over the baseline seq2seq model.
• Adding 8 residual blocks after two initial convolutional layers reduces WER to 11.11%, representing a 24.7% relative improvement over the baseline.
• Replacing standard LSTMs with convolutional LSTMs in residual blocks yields an additional 7% relative WER reduction, reaching 10.53%.
• The model with 15 layers (including 8 residual blocks and NiN modules) outperforms previous state-of-the-art models, which reported 18.0% WER on the same benchmark.
• Batch normalization was essential for training deep models, as it stabilized training and prevented divergence in high-variance attention gradients.
• The use of 3×1 filters in convolutional LSTMs was found effective in capturing temporal dynamics while preserving spectral structure in the hidden state.