[Paper Review] Feature Learning in Deep Neural Networks - Studies on Speech Recognition Tasks

Dong Yu, Michael L. Seltzer+3 more|arXiv (Cornell University)|Jan 16, 2013

Speech Recognition and Synthesis17 references177 citations

TL;DR

This paper demonstrates that deep neural networks (DNNs) learn highly invariant and discriminative features through hierarchical nonlinear transformations, making them robust to small input variations like speaker, noise, and bandwidth differences. Despite requiring no explicit adaptation, DNN-based ASR systems achieve state-of-the-art performance on the Aurora 4 benchmark, outperforming complex GMM-HMM systems that rely on iterative adaptation and normalization.

ABSTRACT

Recent studies have shown that deep neural networks (DNNs) perform significantly better than shallow networks and Gaussian mixture models (GMMs) on large vocabulary speech recognition tasks. In this paper, we argue that the improved accuracy achieved by the DNNs is the result of their ability to extract discriminative internal representations that are robust to the many sources of variability in speech signals. We show that these representations become increasingly insensitive to small perturbations in the input with increasing network depth, which leads to better speech recognition performance with deeper networks. We also show that DNNs cannot extrapolate to test samples that are substantially different from the training examples. If the training data are sufficiently representative, however, internal features learned by the DNN are relatively stable with respect to speaker differences, bandwidth differences, and environment distortion. This enables DNN-based recognizers to perform as well or better than state-of-the-art systems based on GMMs or shallow networks without the need for explicit model adaptation or feature normalization.

Motivation & Objective

• To investigate how deep neural networks (DNNs) learn robust internal representations for speech recognition.
• To analyze whether DNNs generalize well to unseen but similar variations in speech signals, such as speaker differences, noise, and bandwidth changes.
• To evaluate whether DNNs can replace complex adaptation techniques like VTLN, MLLR, or VTS in GMM-based systems.
• To determine the limits of DNN generalization when test data significantly differ from training data.
• To demonstrate that representative training data alone can yield robust performance without explicit model adaptation.

Proposed method

• Interprets DNNs as a stack of log-linear models, combining nonlinear feature transformation with discriminative classification.
• Uses a deep architecture with 7 hidden layers of 2048 units each, trained via layer-wise pre-training followed by discriminative fine-tuning using backpropagation.
• Employs 24-dimensional log mel-filterbank features with dynamic cepstral coefficients, forming a 792-dimensional input from an 11-frame context window.
• Evaluates feature invariance by measuring sensitivity to small input perturbations across network layers.
• Compares DNN performance against multiple GMM-HMM baselines, including systems using MPE, NAT, VTS, and MLLR for adaptation.
• Conducts ablation studies on clean-only and wideband-only training to test generalization limits under severe distributional shift.

Experimental results

Research questions

• RQ1 How do deep neural networks learn internal representations that are robust to small input variations in speech signals?
• RQ2 To what extent can DNNs generalize to test samples that are substantially different from training data?
• RQ3 Can DNNs achieve state-of-the-art performance on speech recognition tasks without explicit model adaptation or feature normalization?
• RQ4 How does network depth affect the invariance of learned features to input perturbations?
• RQ5 What role does training data representativeness play in enabling DNNs to generalize to speaker, noise, and bandwidth variations?

Key findings

• Deeper DNNs learn features that are increasingly insensitive to small input perturbations, with higher-layer representations showing greater invariance.
• The DNN system achieved a 13.4% average word error rate (WER) on the Aurora 4 benchmark, matching the best-reported result without requiring multiple decoding passes or adaptation.
• When trained only on clean speech, the DNN’s performance degraded to 30.6% average WER on noisy and distorted test sets, confirming its inability to extrapolate beyond training distribution.
• A DNN trained only on wideband data failed to recognize narrowband speech, indicating that generalization is limited to small distributional shifts.
• Despite lacking explicit adaptation, the DNN outperformed GMM-HMM systems that used complex techniques like VTS, MLLR, and NAT, achieving comparable or better performance with a single forward pass.
• The DNN learned speaker-invariant and noise-robust features when trained on diverse data, enabling robust performance without speaker or environmental adaptation.