[Paper Review] Residual Convolutional CTC Networks for Automatic Speech Recognition

Yisen Wang, Xuejiao Deng+2 more|arXiv (Cornell University)|Feb 24, 2017

Speech Recognition and SynthesisComputer Science25 references66 citations

TL;DR

The paper introduces a deep and wide RCNN-CTC architecture with residual connections and end-to-end CTC training, plus a CTC-based system combination to boost ASR accuracy on WSJ and Tencent Chat data.

ABSTRACT

Deep learning approaches have been widely used in Automatic Speech Recognition (ASR) and they have achieved a significant accuracy improvement. Especially, Convolutional Neural Networks (CNNs) have been revisited in ASR recently. However, most CNNs used in existing work have less than 10 layers which may not be deep enough to capture all human speech signal information. In this paper, we propose a novel deep and wide CNN architecture denoted as RCNN-CTC, which has residual connections and Connectionist Temporal Classification (CTC) loss function. RCNN-CTC is an end-to-end system which can exploit temporal and spectral structures of speech signals simultaneously. Furthermore, we introduce a CTC-based system combination, which is different from the conventional frame-wise senone-based one. The basic subsystems adopted in the combination are different types and thus mutually complementary to each other. Experimental results show that our proposed single system RCNN-CTC can achieve the lowest word error rate (WER) on WSJ and Tencent Chat data sets, compared to several widely used neural network systems in ASR. In addition, the proposed system combination can offer a further error reduction on these two data sets, resulting in relative WER reductions of $14.91\%$ and $6.52\%$ on WSJ dev93 and Tencent Chat data sets respectively.

Motivation & Objective

• Motivate the need for deeper CNNs in ASR to capture temporal and spectral speech information.
• Develop a very deep and wide CNN architecture with residual connections for ASR using CTC loss.
• Enable end-to-end training without frame-wise alignment by adopting CTC.
• Propose a novel CTC-based system combination to leverage complementary subsystems.
• Demonstrate improved WER on standard and large-scale datasets and analyze system combination benefits.

Proposed method

• Propose RCNN-CTC: a deep (over 40 layers) and wide CNN with residual blocks to model time-frequency structure of speech.
• Use large initial convolution followed by four groups of ResBlocks with small 3x3 filters and batch normalization with ReLU activations.
• Train end-to-end with Connectionist Temporal Classification (CTC) loss to align speech frames to label sequences without pre-aligned data.
• Introduce a CTC-based system combination selecting diverse subsystems (RCNN-CTC, BLSTM, CLDNN) via a maximal correct word rate (MCWR) criterion.
• Align, time-normalize, and construct a word transition network (WTN) across subsystems, then apply a voting scheme (max confidence) to produce final transcripts.
• Analyze 1-best versus N-best hypotheses, finding no improvement from N-best in this setup.

Experimental results

Research questions

• RQ1 Can a very deep and wide residual CNN with CTC outperform conventional CNN/LSTM-based ASR models?
• RQ2 Does end-to-end CTC training combined with residual CNNs yield lower WER on WSJ and large-scale mobile data like Tencent Chat?
• RQ3 Can a CTC-based system combination leveraging heterogeneous subsystems provide additional gains beyond a single RCNN-CTC model?

Key findings

• RCNN-CTC achieves the lowest WER among single-system baselines on WSJ eval92 and dev93 compared to VGG+CTC, CLDNN+CTC, and BLSTM+CTC.
• The proposed CTC-based system combination yields further WER reductions, achieving up to 14.91% relative improvement on WSJ dev93 and 6.52% on Tencent Chat over the best single system.
• On Tencent Chat, RCNN-CTC outperforms other single systems (VGG, CLDNN, BLSTM) in WER, highlighting the benefit of deep CNN with residual connections on large data.
• MCWR-based subsystems selection favors combining RCNN-CTC with BLSTM and CLDNN, achieving the best combined WER in WSJ and Tencent Chat experiments.
• Using N-best hypotheses does not improve over 1-best in the proposed system combination.