[Paper Review] Environmental Noise Embeddings for Robust Speech Recognition

Suyoun Kim, Bhiksha Raj+1 more|arXiv (Cornell University)|Jan 11, 2016

Speech Recognition and Synthesis21 references22 citations

TL;DR

This paper proposes a deep neural network framework that learns environmental noise embeddings to improve robustness in speech recognition. By training a separate DNN to extract discriminative noise embeddings from noisy speech and concatenating them with acoustic features, the method significantly reduces word error rate (WER) across diverse noise conditions, outperforming baseline, i-vector, NAT, and multi-task learning approaches, especially in unseen and low SNR environments.

ABSTRACT

We propose a novel deep neural network architecture for speech recognition that explicitly employs knowledge of the background environmental noise within a deep neural network acoustic model. A deep neural network is used to predict the acoustic environment in which the system in being used. The discriminative embedding generated at the bottleneck layer of this network is then concatenated with traditional acoustic features as input to a deep neural network acoustic model. Through a series of experiments on Resource Management, CHiME-3 task, and Aurora4, we show that the proposed approach significantly improves speech recognition accuracy in noisy and highly reverberant environments, outperforming multi-condition training, noise-aware training, i-vector framework, and multi-task learning on both in-domain noise and unseen noise.

Motivation & Objective

• Address performance degradation in automatic speech recognition (ASR) due to mismatched training and test environments, particularly background noise and reverberation.
• Overcome limitations of existing noise adaptation techniques such as fixed noise estimates in Noise-Aware Training (NAT) and poor generalization of i-vector-based methods in unseen noise.
• Develop a dynamic, end-to-end trainable noise adaptation framework that explicitly models environmental acoustics using deep neural networks.
• Improve robustness in both in-domain and unseen noise conditions, including highly reverberant and low signal-to-noise ratio (SNR) environments.
• Demonstrate that noise embeddings can generalize better than traditional noise-aware or i-vector-based methods across diverse datasets like CHiME-3 and Aurora4.

Proposed method

• Train a separate deep neural network (N_DNN) to predict the environmental noise from raw speech features, using a bottleneck layer to generate a discriminative noise embedding.
• Extract a low-dimensional noise embedding from the bottleneck layer of the N_DNN, which captures the acoustic characteristics of the background noise.
• Concatenate the learned noise embedding with standard acoustic features (e.g., MFCCs) before feeding them into the main DNN acoustic model for speech recognition.
• Train the full system in a two-stage process: first train the noise embedding network, then fine-tune the end-to-end ASR model with the concatenated features.
• Compare the proposed method with alternative approaches: Noise-Aware Training (NAT), i-vector-based noise modeling (N_GMM), and multi-task learning (MTL).
• Use 40-dimensional noise embeddings for all experiments, with training data limited to four noise types to evaluate generalization to unseen noise.

Experimental results

Research questions

• RQ1 Can a discriminatively trained deep neural network learn meaningful, low-dimensional noise embeddings that improve ASR robustness in noisy environments?
• RQ2 How does the proposed noise embedding method compare to existing techniques like NAT, i-vector, and multi-task learning in terms of WER reduction across in-domain and unseen noise conditions?
• RQ3 Does the use of noise embeddings lead to consistent performance gains across diverse datasets such as CHiME-3 and Aurora4, especially at low SNR levels?
• RQ4 To what extent does the model generalize to unseen noise types when only four noise types are used during training?
• RQ5 Can noise embeddings improve recognition accuracy even in clean test conditions, indicating improved generalization beyond noise-specific adaptation?

Key findings

• The +N_DNN model achieved a 2.92% relative WER improvement over the baseline on the Aurora4 dataset with background music at 0 dB SNR, while only a 0.19% improvement on the clean set, indicating strong noise-specific gains.
• On the CHiME-3 task, +N_DNN reduced WER by 0.3% (from 15.6% to 15.3%) in the in-domain noise condition and by 0.2% (from 11.7% to 11.5%) in the unseen noise condition, with statistical significance (p < 0.05).
• The +N_DNN model outperformed both +N_NAT and +N_GMM across all test sets, with +N_NAT showing only marginal gains and +N_GMM performing worse than the baseline on unseen noise.
• Visualizations of feature embeddings showed that +N_DNN produced significantly more discriminative input features across different noise environments, especially in unseen noise cases.
• The method achieved a 2.2% relative WER reduction compared to baseline on in-domain noise and a 0.9% relative reduction on unseen noise, demonstrating strong generalization capability.
• The performance gain was most pronounced at low SNRs, confirming that noise embeddings are particularly effective in challenging acoustic conditions.