[Paper Review] Learning Noise-Invariant Representations for Robust Speech Recognition

Davis Liang, Zhiheng Huang+1 more|arXiv (Cornell University)|Jul 17, 2018

Speech Recognition and Synthesis28 references4 citations

TL;DR

This paper proposes Invariant-Representation Learning (IRL), a regularization method that encourages deep neural networks to produce identical hidden representations for clean speech and their noisy counterparts during training. By penalizing the distance between representations at intermediate layers using L2 and cosine similarity, IRL reduces character error rates by over 45% on clean and other Librispeech test sets, significantly outperforming data augmentation and prior baselines, especially under out-of-domain noise.

ABSTRACT

Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs. Small amounts of noise can destroy the performance of an otherwise state-of-the-art model. To harden models against background noise, practitioners often perform data augmentation, adding artificially-noised examples to the training set, carrying over the original label. In this paper, we hypothesize that a clean example and its superficially perturbed counterparts shouldn't merely map to the same class --- they should map to the same representation. We propose invariant-representation-learning (IRL): At each training iteration, for each training example,we sample a noisy counterpart. We then apply a penalty term to coerce matched representations at each layer (above some chosen layer). Our key results, demonstrated on the Librispeech dataset are the following: (i) IRL significantly reduces character error rates (CER) on both 'clean' (3.3% vs 6.5%) and 'other' (11.0% vs 18.1%) test sets; (ii) on several out-of-domain noise settings (different from those seen during training), IRL's benefits are even more pronounced. Careful ablations confirm that our results are not simply due to shrinking activations at the chosen layers.

Motivation & Objective

• Address the brittleness of state-of-the-art ASR models to input noise, which severely degrades performance even with small perturbations.
• Overcome the limitations of standard data augmentation, which only encourages class-level invariance but not representation-level invariance.
• Develop a method that enforces identical hidden representations for clean and noisy versions of the same input, improving robustness without sacrificing inference speed.
• Demonstrate that representation-level invariance leads to stronger generalization than logit-level invariance or standard regularization techniques.

Proposed method

• At each training iteration, for each clean input, a noisy counterpart is synthesized by adding a random noise snippet (from MUSAN) at a random volume to the original waveform.
• The model processes both clean and noisy inputs through the same network, producing hidden representations at each layer.
• A regularization loss is applied to minimize the L2 and cosine distance between corresponding hidden representations of clean and noisy pairs.
• The total loss combines standard cross-entropy loss on predictions and the representation-invariance penalty, encouraging the network to learn noise-robust features.
• The invariance penalty is applied only above a chosen layer (e.g., encoder layer), allowing early layers to remain sensitive to input variations.
• The method is evaluated on Librispeech using a sequence-to-sequence model with cross-entropy loss, without language modeling, to isolate the effect of IRL.

Figure 1 : Diagram demonstrating the various terms in the IRL loss function as applied to a Seq2Seq attention model. Dotted lines represent shared weights.

Experimental results

Research questions

• RQ1 Can enforcing identical hidden representations for clean and noisy speech inputs improve robustness in automatic speech recognition?
• RQ2 Does representation-level invariance outperform logit-level invariance or standard data augmentation in handling out-of-domain noise?
• RQ3 Is the performance gain from IRL due to shrinking activations or due to actual representation alignment?
• RQ4 How does IRL compare to adversarial training and other regularization techniques in terms of generalization under diverse noise conditions?

Key findings

• IRL reduces character error rate (CER) to 3.3% on the Librispeech test-clean set and 11.0% on test-other, compared to 6.5% and 18.1% for the baseline, representing a 49% and 40% relative improvement respectively.
• On out-of-domain noise such as reverberation, IRL-C achieves 13.8% CER versus 21.0% for data-augmented models and 24.1% for the baseline, showing strong robustness gains.
• Under overlapping speech noise, IRL-C achieves 14.1% CER, significantly outperforming the baseline (91.5%) and data-augmented model (32.0%).
• For low signal-to-noise ratio (SNR) conditions, IRL-C achieves 5.7% CER, compared to 10.8% for the baseline and 27.8% for data-augmented models.
• The IRL model maintains strong performance on resampled telephonic data (6.4% CER) compared to baseline (14.2%) and data-augmented (12.2%) models.
• Empirical analysis confirms that IRL reduces both L2 and cosine distances between clean and noisy representations, with IRL-C maintaining close alignment across all layers, unlike IRL-E which diverges after the encoder layer.

(a) L2 distance