[Paper Review] Recent Progresses in Deep Learning based Acoustic Models (Updated)

Dong Yu, Jinyu Li|arXiv (Cornell University)|Apr 25, 2018

Speech Recognition and Synthesis156 references3 citations

TL;DR

This paper reviews recent advances in deep learning-based acoustic models for automatic speech recognition, focusing on models that exploit variable-length context (e.g., RNNs, CNNs), end-to-end architectures (CTC, attention-based seq2seq), robustness techniques (adaptation, speech enhancement), and efficient decoding (knowledge distillation, quantization). It highlights improved performance through joint optimization and robust training, especially in far-field and low-resource settings.

ABSTRACT

In this paper, we summarize recent progresses made in deep learning based acoustic models and the motivation and insights behind the surveyed techniques. We first discuss acoustic models that can effectively exploit variable-length contextual information, such as recurrent neural networks (RNNs), convolutional neural networks (CNNs), and their various combination with other models. We then describe acoustic models that are optimized end-to-end with emphasis on feature representations learned jointly with rest of the system, the connectionist temporal classification (CTC) criterion, and the attention-based sequence-to-sequence model. We further illustrate robustness issues in speech recognition systems, and discuss acoustic model adaptation, speech enhancement and separation, and robust training strategies. We also cover modeling techniques that lead to more efficient decoding and discuss possible future directions in acoustic model research.

Motivation & Objective

• To summarize recent progress in deep learning-based acoustic models for automatic speech recognition (ASR), particularly post-2015.
• To analyze the motivations and core mechanisms behind key architectures such as RNNs, CNNs, CTC, and attention-based models.
• To examine techniques that improve robustness in real-world conditions, including noise, reverberation, and speaker variability.
• To explore methods for efficient decoding, such as model compression and quantization, to enable real-time deployment.
• To identify open challenges and future research directions in acoustic modeling, especially for far-field and low-resource ASR.

Proposed method

• Utilizes recurrent neural networks (RNNs), particularly Long Short-Term Memory (LSTM) units, to model long-range temporal dependencies in speech sequences by maintaining hidden states that encode history.
• Employs convolutional neural networks (CNNs) with local receptive fields to extract hierarchical contextual features from speech input, enabling effective modeling of variable-length context.
• Applies the Connectionist Temporal Classification (CTC) criterion to train end-to-end models that map raw or high-level features directly to transcribed sequences without requiring forced alignment.
• Introduces attention-based sequence-to-sequence models that allow alignment-free, joint optimization of acoustic and language modeling components through soft attention mechanisms.
• Employs permutation invariant training (PIT) to improve speech separation performance in multi-speaker scenarios by minimizing permutation ambiguity during loss computation.
• Uses knowledge distillation and quantization techniques to compress large models into smaller, faster-decoding variants while preserving accuracy.

Experimental results

Research questions

• RQ1 How can acoustic models better exploit variable-length contextual information compared to fixed-context DNNs?
• RQ2 What are the advantages and limitations of end-to-end models (CTC and attention-based) in joint acoustic and language modeling?
• RQ3 How can speech enhancement and separation techniques be integrated into acoustic modeling to improve robustness in noisy or multi-speaker environments?
• RQ4 What strategies can effectively reduce inference cost while maintaining recognition accuracy in real-time ASR systems?
• RQ5 How can models be designed to adapt dynamically to new speakers or environments with limited data?

Key findings

• LSTM and CNN-based hybrid models significantly outperform traditional DNN/HMM systems by effectively modeling long-range dependencies and variable-length context.
• End-to-end models using CTC and attention mechanisms achieve state-of-the-art performance when trained on large-scale datasets (e.g., hundreds of thousands of hours), demonstrating the effectiveness of joint optimization.
• Permutation Invariant Training (PIT) improves multi-speaker speech separation, especially in same-gender scenarios, by enabling robust loss computation across speaker permutations.
• Knowledge distillation and model quantization reduce computational cost by up to 1/3 while maintaining high accuracy, enabling efficient deployment on edge devices.
• Robust training strategies, including data augmentation and domain adaptation, help mitigate performance degradation due to training-test distribution shifts.
• Future models that jointly optimize language modeling with acoustic modeling—especially using syllable-level units—may enable easier and more scalable integration of new vocabulary items.