[Paper Review] Deep Recurrent Convolutional Neural Network: Improving Performance For Speech Recognition

Zewang Zhang, Zheng Sun+4 more|arXiv (Cornell University)|Nov 22, 2016

Speech Recognition and Synthesis37 references17 citations

TL;DR

This paper proposes a novel deep recurrent convolutional neural network (RCNN) architecture that reverses the traditional CNN-RNN order, placing recurrent networks before convolutional layers to improve training efficiency and recognition accuracy in speech recognition. By integrating deep residual learning, the model achieves faster convergence and a state-of-the-art phoneme error rate (PER) of 17.33% on the TIMIT dataset, outperforming both standard and residual-enhanced baseline models.

ABSTRACT

A deep learning approach has been widely applied in sequence modeling problems. In terms of automatic speech recognition (ASR), its performance has significantly been improved by increasing large speech corpus and deeper neural network. Especially, recurrent neural network and deep convolutional neural network have been applied in ASR successfully. Given the arising problem of training speed, we build a novel deep recurrent convolutional network for acoustic modeling and then apply deep residual learning to it. Our experiments show that it has not only faster convergence speed but better recognition accuracy over traditional deep convolutional recurrent network. In the experiments, we compare the convergence speed of our novel deep recurrent convolutional networks and traditional deep convolutional recurrent networks. With faster convergence speed, our novel deep recurrent convolutional networks can reach the comparable performance. We further show that applying deep residual learning can boost the convergence speed of our novel deep recurret convolutional networks. Finally, we evaluate all our experimental networks by phoneme error rate (PER) with our proposed bidirectional statistical n-gram language model. Our evaluation results show that our newly proposed deep recurrent convolutional network applied with deep residual learning can reach the best PER of 17.33\% with the fastest convergence speed on TIMIT database. The outstanding performance of our novel deep recurrent convolutional neural network with deep residual learning indicates that it can be potentially adopted in other sequential problems.

Motivation & Objective

• To address the slow convergence and high training time of traditional deep convolutional recurrent networks (CRNNs) in automatic speech recognition (ASR).
• To improve recognition accuracy and training efficiency by reordering the network architecture, placing recurrent layers before convolutional layers.
• To investigate the effectiveness of deep residual learning in enhancing both convergence speed and recognition performance in the proposed architecture.
• To evaluate the proposed model using a bidirectional statistical n-gram language model and compare it against standard CRNNs and residual variants.

Proposed method

• Proposes a new architecture, RCNN, where recurrent layers are placed before convolutional layers, reversing the typical CRNN structure.
• Applies residual learning by adding shortcut connections as identity mappings to stabilize training and accelerate convergence.
• Uses connectionist temporal classification (CTC) for end-to-end training without requiring pre-segmented data.
• Employs a bidirectional statistical n-gram language model trained on the full training set to decode and refine phoneme sequences.
• Trains models using mini-batch stochastic gradient descent with batch normalization and ReLU activation functions.
• Compares multiple variants: CR1–CR4 (standard CRNNs), RC1–RC6 (proposed RCNNs), and Res-RC2/Res-CR2 (residual versions).

Experimental results

Research questions

• RQ1 Does reordering the network architecture—placing recurrent layers before convolutional layers—improve convergence speed and recognition accuracy in ASR?
• RQ2 Can deep residual learning significantly accelerate convergence and improve performance in the proposed recurrent convolutional network architecture?
• RQ3 How does the proposed RCNN architecture compare to standard CRNNs in terms of phoneme error rate and training efficiency?
• RQ4 What causes degradation in performance when applying residual blocks to deeper CRNNs, and how can this be mitigated?

Key findings

• The proposed RC2 model achieves a test phoneme error rate (PER) of 20.71%, showing competitive performance compared to traditional deep belief networks.
• The CR2 model achieves a test PER of 18.73%, outperforming RC2, but requires longer training time.
• The Res-RC2 model achieves the best test PER of 17.33% with the fastest convergence speed, demonstrating the effectiveness of residual learning in the new architecture.
• The Res-CR2 model shows only a slight improvement (18.90% PER) over CR2, likely due to the depth of the residual block restricting performance gains.
• The RCNN architecture converges faster than standard CRNNs in the first half of training, indicating improved optimization dynamics.
• The degradation observed in Res-CR2 is attributed to the extreme depth of its residual block and the influence of top recurrent layers on convergence.