[Paper Review] Frustratingly Easy Noise-aware Training of Acoustic Models

Desh Raj, Jesús Villalba+2 more|arXiv (Cornell University)|Nov 4, 2020

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper proposes a simple yet effective noise-aware training method for acoustic models by using utterance-level noise vectors derived from the mean of speech and silence frames, leveraging existing GMM-HMM alignments without extra computation. The approach achieves 6–7% relative WER improvement on AMI and Aurora-4 datasets, outperforming prior embedding-based baselines and enabling online streaming ASR via frame-level maximum likelihood estimation of noise vectors.

ABSTRACT

Environmental noises and reverberation have a detrimental effect on the performance of automatic speech recognition (ASR) systems. Multi-condition training of neural network-based acoustic models is used to deal with this problem, but it requires many-folds data augmentation, resulting in increased training time. In this paper, we propose utterance-level noise vectors for noise-aware training of acoustic models in hybrid ASR. Our noise vectors are obtained by combining the means of speech frames and silence frames in the utterance, where the speech/silence labels may be obtained from a GMM-HMM model trained for ASR alignments, such that no extra computation is required beyond averaging of feature vectors. We show through experiments on AMI and Aurora-4 that this simple adaptation technique can result in 6-7% relative WER improvement. We implement several embedding-based adaptation baselines proposed in literature, and show that our method outperforms them on both the datasets. Finally, we extend our method to the online ASR setting by using frame-level maximum likelihood for the mean estimation.

Motivation & Objective

• To improve robustness of DNN-based acoustic models in noisy environments without increasing training complexity.
• To develop a low-cost, effective noise-aware training technique that reuses existing GMM-HMM alignments for speech activity detection.
• To enable online streaming ASR by deriving real-time noise vector estimates using frame-level maximum likelihood.
• To outperform existing embedding-based adaptation methods such as i-vectors and e-vectors in noise-robust ASR.

Proposed method

• Noise vectors are computed as the mean of speech frames and silence frames in an utterance using SAD labels from pre-trained GMM-HMM alignments.
• The method appends these noise vectors to input features, enabling the DNN to learn non-linear transformations for noise robustness.
• For online ASR, noise vector means are estimated using a probabilistic model with maximum likelihood estimation over streaming frames.
• Scaling factors for speech and noise components are learned via EM optimization, initialized using global statistics or default values when no frames are observed.
• The approach avoids front-end signal processing and instead embeds noise awareness directly into the acoustic model via structured input conditioning.
• The method is compatible with sequence discriminative training criteria like lattice-free MMI, using the same alignments used for acoustic model training.

Experimental results

Research questions

• RQ1 Can a simple, computationally lightweight noise vector derived from speech and silence frame means improve acoustic model robustness?
• RQ2 Does this noise-aware training approach outperform existing embedding-based adaptation methods like i-vectors and e-vectors?
• RQ3 Can the noise vector estimation be adapted for online streaming ASR without waiting for full utterance completion?
• RQ4 How does the performance of this method compare to multi-condition training with extensive data augmentation?

Key findings

• The proposed method achieves a 6–7% relative WER improvement on the AMI and Aurora-4 datasets compared to baseline models.
• The method outperforms established baselines such as i-vectors and e-vectors in noise-robust ASR, demonstrating superior generalization to environmental distortions.
• The noise vector estimation using speech and silence frame means provides a more informative noise representation than prior methods that use only start/end frames.
• The online adaptation version using frame-level maximum likelihood estimation enables real-time noise-aware inference with minimal latency.
• The method incurs no additional computational cost beyond standard GMM-HMM alignment, making it practical for large-scale deployment.