[Paper Review] On Cross-Corpus Generalization of Deep Learning Based Speech Enhancement

Ashutosh Pandey, DeLiang Wang|arXiv (Cornell University)|Feb 10, 2020

Speech and Audio Processing40 references5 citations

TL;DR

This paper identifies channel mismatch as the primary cause of poor cross-corpus generalization in deep learning-based speech enhancement, especially at low SNR. It proposes two key solutions: training on diverse, crowd-sourced corpora like LibriAll and using a smaller STFT frame shift, which together significantly improve performance on untrained corpora, with STOI gains up to 8.1% on IEEE Male at -5 dB SNR.

ABSTRACT

In recent years, supervised approaches using deep neural networks (DNNs) have become the mainstream for speech enhancement. It has been established that DNNs generalize well to untrained noises and speakers if trained using a large number of noises and speakers. However, we find that DNNs fail to generalize to new speech corpora in low signal-to-noise ratio (SNR) conditions. In this work, we establish that the lack of generalization is mainly due to the channel mismatch, i.e. different recording conditions between the trained and untrained corpus. Additionally, we observe that traditional channel normalization techniques are not effective in improving cross-corpus generalization. Further, we evaluate publicly available datasets that are promising for generalization. We find one particular corpus to be significantly better than others. Finally, we find that using a smaller frame shift in short-time processing of speech can significantly improve cross-corpus generalization. The proposed techniques to address cross-corpus generalization include channel normalization, better training corpus, and smaller frame shift in short-time Fourier transform (STFT). These techniques together improve the objective intelligibility and quality scores on untrained corpora significantly.

Motivation & Objective

• To investigate why deep learning-based speech enhancement models fail to generalize to untrained speech corpora, especially under low SNR conditions.
• To identify the root cause of poor cross-corpus generalization, particularly in low SNR regimes.
• To evaluate the effectiveness of traditional channel normalization techniques (e.g., CMS, RASTA) in improving cross-corpus generalization.
• To assess the suitability of publicly available datasets for enhancing cross-corpus generalization.
• To propose and validate new techniques—specifically, using the LibriAll corpus and reducing STFT frame shift—for improving robustness across diverse recording conditions.

Proposed method

• The study evaluates deep neural network (DNN) models trained on various corpora, including LibriSpeech, VoxCeleb, TIMIT, and IEEE, and tests generalization on untrained corpora.
• It applies channel normalization techniques—cepstral mean subtraction (CMS) and RASTA filtering—in the log-spectrum domain to mitigate channel mismatch effects.
• The authors introduce a smaller STFT frame shift (4 ms) in the short-time processing pipeline to improve temporal resolution and model robustness.
• A modified loss function is used to enhance model generalization, though its effectiveness is limited in low SNR conditions.
• The LibriAll corpus, derived from crowd-sourced recordings across diverse acoustic environments, is evaluated as a superior training resource.
• Models are trained on anechoic data and tested on noisy-reverberant data using real room impulse responses (RIRs) to assess robustness.

Experimental results

Research questions

• RQ1 Why do DNN-based speech enhancement models fail to generalize to untrained speech corpora, particularly at low SNR levels?
• RQ2 To what extent does channel mismatch between training and test corpora contribute to the generalization gap in speech enhancement?
• RQ3 How effective are traditional channel normalization techniques (e.g., CMS, RASTA) in improving cross-corpus generalization in supervised speech enhancement?
• RQ4 Which publicly available speech corpora are most effective for improving cross-corpus generalization, and why?
• RQ5 Can reducing the STFT frame shift improve cross-corpus generalization, and how does it compare to corpus selection in performance gains?

Key findings

• The primary cause of poor cross-corpus generalization is channel mismatch between training and test corpora, especially pronounced at low SNR levels.
• Traditional channel normalization techniques (CMS and RASTA) improve performance on untrained corpora but offer only limited gains, particularly in low SNR conditions.
• Using the LibriAll corpus for training improves STOI by 8.1% on IEEE Male and 5.3% on IEEE Female at -5 dB SNR, significantly outperforming other corpora like VoxCeleb.
• Reducing the STFT frame shift to 4 ms improves STOI by 3.5% on TIMIT, 1.8% on IEEE Male, and 3.5% on IEEE Female at -5 dB SNR, demonstrating strong gains.
• The combination of LibriAll and 4 ms frame shift achieves STOI scores of 70.8% on WSJ and 63.7% on IEEE Female at -5 dB, approaching the performance of same-corpus training (62.8% and 65.3% respectively).
• In noisy-reverberant conditions, the model trained with LibriAll and 4 ms frame shift achieves the best performance, with STOI of 70.3% on IEEE Female and PESQ of 1.94, showing robustness to reverberation and noise.