[Paper Review] An Investigation of End-to-End Multichannel Speech Recognition for Reverberant and Mismatch Conditions

Aswin Shanmugam Subramanian, Xiaofei Wang+4 more|arXiv (Cornell University)|Apr 19, 2019

Speech and Audio Processing34 references17 citations

TL;DR

This paper proposes an end-to-end multichannel ASR system that jointly optimizes neural dereverberation (WPE), beamforming (MVDR), and acoustic modeling using only ASR objective loss, without requiring parallel clean-reverberant data or iterative inference. It achieves state-of-the-art performance on REVERB and significant gains on mismatched DIRHA data, demonstrating robustness to real-world reverberant and noisy conditions through joint, differentiable training of all components.

ABSTRACT

Sequence-to-sequence (S2S) modeling is becoming a popular paradigm for automatic speech recognition (ASR) because of its ability to jointly optimize all the conventional ASR components in an end-to-end (E2E) fashion. This report investigates the ability of E2E ASR from standard close-talk to far-field applications by encompassing entire multichannel speech enhancement and ASR components within the S2S model. There have been previous studies on jointly optimizing neural beamforming alongside E2E ASR for denoising. It is clear from both recent challenge outcomes and successful products that far-field systems would be incomplete without solving both denoising and dereverberation simultaneously. This report uses a recently developed architecture for far-field ASR by composing neural extensions of dereverberation and beamforming modules with the S2S ASR module as a single differentiable neural network and also clearly defining the role of each subnetwork. The original implementation of this architecture was successfully applied to the noisy speech recognition task (CHiME-4), while we applied this implementation to noisy reverberant tasks (DIRHA and REVERB). Our investigation shows that the method achieves better performance than conventional pipeline methods on the DIRHA English dataset and comparable performance on the REVERB dataset. It also has additional advantages of being neither iterative nor requiring parallel noisy and clean speech data.

Motivation & Objective

• To investigate the feasibility of integrating multichannel speech enhancement (dereverberation and beamforming) directly into end-to-end ASR for far-field applications.
• To evaluate whether joint optimization of dereverberation, beamforming, and ASR via a single differentiable loss improves robustness in reverberant and mismatched conditions.
• To eliminate reliance on parallel clean-reverberant speech data and iterative processing in end-to-end multichannel ASR.
• To assess generalization performance on real-world far-field data (e.g., DIRHA, REVERB real sets) where training data conditions differ.

Proposed method

• The model integrates neural WPE-based dereverberation and MVDR beamforming as differentiable subnetworks within a sequence-to-sequence ASR framework.
• The dereverberation subnetwork uses a time-frequency mask to estimate the magnitude spectrum of the desired signal via a DNN, replacing the iterative WPE process.
• The beamforming subnetwork uses a differentiable MVDR filter computed from estimated power spectral density matrices, with optional speech activity detection (SAD)-type masks.
• All components are trained jointly using only the ASR objective, with no need for parallel clean data or explicit signal-level supervision.
• During training, the model randomly skips either dereverberation or beamforming branches to improve robustness and generalization.
• The system supports arbitrary numbers of input channels and generates enhanced spectrograms from intermediate subnetwork outputs.

Experimental results

Research questions

• RQ1 Can end-to-end training of a multichannel ASR system with integrated neural dereverberation and beamforming outperform conventional pipeline methods in reverberant and noisy environments?
• RQ2 Does joint optimization of dereverberation and beamforming via the ASR objective alone yield robust performance on mismatched real-world test sets?
• RQ3 How does the use of different mask types (TF vs. SAD) affect performance in challenging far-field conditions?
• RQ4 Can the system generalize to real far-field data without requiring parallel clean-reverberant training data?

Key findings

• The E2E model with joint dereverberation and beamforming using SAD masks achieved a 5.2% relative WER improvement over the best pipeline method on the mismatched DIRHA test set.
• On the REVERB simulation sets, the joint E2E model with TF masks achieved the best performance, outperforming both pipeline and standalone E2E baselines.
• The model achieved comparable or better performance than state-of-the-art systems on the REVERB challenge's official test sets, particularly on real far-field conditions.
• The system generalizes well to unseen channel configurations and does not require iterative processing or parallel clean data.
• Visual analysis of spectrograms confirmed that the enhanced output exhibits reduced smearing and clearer background, indicating effective dereverberation and denoising.
• The method successfully performs joint optimization of all components using only the ASR objective, demonstrating that signal-level objectives are not necessary for effective enhancement.