[Paper Review] End-to-End Dereverberation, Beamforming, and Speech Recognition with Improved Numerical Stability and Advanced Frontend

Wangyou Zhang, Christoph Boeddeker+8 more|arXiv (Cornell University)|Feb 23, 2021

Speech and Audio Processing47 references33 citations

TL;DR

This paper proposes an end-to-end framework for joint dereverberation, beamforming, and speech recognition in multichannel, multi-speaker, reverberant environments. By introducing four numerical stabilization techniques—diagonal loading, mask flooring, stable complex operations, and double-precision computation—alongside VAD-like masks to mitigate frequency permutation, the system achieves 35% relative WER reduction over conventional E2E ASR and strong speech enhancement performance (SDR = 12.5 dB) using only ASR supervision.

ABSTRACT

Recently, the end-to-end approach has been successfully applied to multi-speaker speech separation and recognition in both single-channel and multichannel conditions. However, severe performance degradation is still observed in the reverberant and noisy scenarios, and there is still a large performance gap between anechoic and reverberant conditions. In this work, we focus on the multichannel multi-speaker reverberant condition, and propose to extend our previous framework for end-to-end dereverberation, beamforming, and speech recognition with improved numerical stability and advanced frontend subnetworks including voice activity detection like masks. The techniques significantly stabilize the end-to-end training process. The experiments on the spatialized wsj1-2mix corpus show that the proposed system achieves about 35% WER relative reduction compared to our conventional multi-channel E2E ASR system, and also obtains decent speech dereverberation and separation performance (SDR=12.5 dB) in the reverberant multi-speaker condition while trained only with the ASR criterion.

Motivation & Objective

• Address the severe performance degradation of end-to-end ASR in reverberant and noisy multi-speaker environments.
• Stabilize the training of joint end-to-end dereverberation, beamforming, and ASR by resolving numerical instability in WPE and beamformer operations.
• Improve speech enhancement and recognition performance in challenging cocktail party scenarios with only transcriptions for training.
• Mitigate the frequency permutation problem in end-to-end training using VAD-like masks for WPE and beamforming.
• Demonstrate that a single E2E system can achieve strong performance in dereverberation, separation, and recognition without separate frontend training.

Proposed method

• Introduce four numerical stabilization techniques: diagonal loading, mask flooring, stable complex operations, and double-precision computation to stabilize matrix inversion in WPE and beamforming.
• Use a shared mask estimator (MaskNet) to generate WPE and beamformer masks, enabling joint optimization.
• Implement a factorized frontend with DNN-WPE for dereverberation and MVDR/wMPDR beamformers for spatial filtering.
• Apply VAD-like masks to suppress non-speech components and reduce frequency permutation during end-to-end training.
• Use a joint CTC/attention-based ASR backend trained end-to-end with the frontend using only transcriptions.
• Optimize the entire system via a PIT-based loss function that aligns predicted and target speech representations.

Experimental results

Research questions

• RQ1 Can numerical instability in WPE and beamformer operations be effectively mitigated in end-to-end training of dereverberation and ASR?
• RQ2 Does the use of VAD-like masks reduce frequency permutation and improve speech enhancement and recognition performance?
• RQ3 Can a single end-to-end system jointly optimize dereverberation, beamforming, and ASR with only ASR supervision?
• RQ4 How do different beamformer variants (MVDR vs. wMPDR) compare in end-to-end training under reverberant conditions?
• RQ5 What is the impact of the proposed stabilization techniques on WER, SDR, PESQ, and STOI metrics in multi-speaker reverberant scenarios?

Key findings

• The proposed system achieves a 35% relative WER reduction compared to the conventional multi-channel end-to-end ASR system on the spatialized wsj1-2mix corpus.
• The system achieves a speech enhancement SDR of 12.5 dB and PESQ of 1.97, indicating strong dereverberation and separation performance.
• The combination of all four stabilization techniques (diagonal loading, mask flooring, stable complex ops, double precision) yields the best convergence and performance.
• The VAD-like mask significantly improves PESQ (1.95), STOI (0.86), and SDR (12.54 dB), reducing frequency permutation effects.
• The wMPDR beamformer with VAD-like masks achieves the best WER of 9.44% and SDR of 8.49 dB, outperforming MVDR-based models.
• The proposed framework with VAD-like masks and stabilization techniques achieves near-topline performance (9.45% WER) compared to the clean WSJ eval92 baseline (4.4% WER), demonstrating strong generalization.