[Paper Review] Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

Dario Amodei, Rishita Anubhai+32 more|arXiv (Cornell University)|Dec 8, 2015

Speech Recognition and Synthesis51 references2,175 citations

TL;DR

Deep Speech 2 demonstrates end-to-end deep learning for English and Mandarin ASR, achieving competitive accuracy with large-scale data and HPC-accelerated training, and deployable online with low latency.

ABSTRACT

We show that an end-to-end deep learning approach can be used to recognize either English or Mandarin Chinese speech--two vastly different languages. Because it replaces entire pipelines of hand-engineered components with neural networks, end-to-end learning allows us to handle a diverse variety of speech including noisy environments, accents and different languages. Key to our approach is our application of HPC techniques, resulting in a 7x speedup over our previous system. Because of this efficiency, experiments that previously took weeks now run in days. This enables us to iterate more quickly to identify superior architectures and algorithms. As a result, in several cases, our system is competitive with the transcription of human workers when benchmarked on standard datasets. Finally, using a technique called Batch Dispatch with GPUs in the data center, we show that our system can be inexpensively deployed in an online setting, delivering low latency when serving users at scale.

Motivation & Objective

• Demonstrate end-to-end ASR that works across English and Mandarin with minimal language-specific engineering.
• Investigate deep architectures (convolutional and recurrent layers) and training techniques to improve accuracy.
• Leverage large-scale labeled data and HPC optimizations to enable rapid experimentation and model deployment.
• Evaluate performance against standard benchmarks and human transcription, and explore production deployment considerations.

Proposed method

• Use an end-to-end RNN-CTC framework to map spectrogram inputs to grapheme outputs.
• Explore architectures with multiple convolutional and bidirectional recurrent layers, including batch normalization for deep RNNs.
• Apply a curriculum-based SortaGrad training strategy to stabilize training on variable-length utterances.
• Experiment with GRU versus simple RNN cells to trade-off accuracy and training efficiency.
• Incorporate frequency-time convolutions and striding to reduce time-steps and computation.
• Introduce unidirectional row-convolution for low-latency online decoding and BatchNorm-enabled deployment strategies.

Experimental results

Research questions

• RQ1 Can end-to-end deep networks trained with CTC achieve competitive WER/CER on English and Mandarin without hand-engineered components?
• RQ2 How do network depth, convolutional strategies, and recurrent unit type affect recognition accuracy in English and Mandarin?
• RQ3 What data scale and HPC optimizations are required to train such models within feasible timeframes?
• RQ4 What deployment strategies (batch dispatch, unidirectional models) enable low-latency online recognition at scale?

Key findings

• English WER reductions up to 43% over prior end-to-end systems on several benchmarks.
• English training data of 11,940 hours and Mandarin data of 9,400 hours used to train large, deep models.
• Training sustains about 50 teraFLOP/s on 16 GPUs, enabling 3–5 day experiments and rapid iteration.
• Batch normalization with sequence-wise statistics accelerates training and improves generalization for deep RNNs.
• GRU cells outperform simple RNNs at equivalent parameter counts, but for very large models simple RNNs can be competitive under fixed compute budgets.
• Mandarin deployment with BatchNorm and row-convolution achieves low latency (98th percentile compute latency 67 ms with 10 simultaneous streams).
• Use of data augmentation and synthesis to further scale effective labeled data for English and Mandarin.