[Paper Review] Dereverberation using joint estimation of dry speech signal and acoustic system

Sanna Wager, Keunwoo Choi+1 more|arXiv (Cornell University)|Jul 24, 2020

Speech and Audio Processing4 citations

TL;DR

This paper proposes a joint deep learning framework for speech dereverberation that simultaneously estimates the dry speech signal and the room impulse response (RIR) from a reverberant input using magnitude STFT representations. The approach combines a U-Net for direct dry speech estimation and a convolutional model for RIR estimation, with shared training objectives to improve accuracy over separate models, achieving state-of-the-art performance on the dry speech task while identifying limitations in RIR precision that motivate future phase-aware and sequential modeling approaches.

ABSTRACT

The purpose of speech dereverberation is to remove quality-degrading effects of a time-invariant impulse response filter from the signal. In this report, we describe an approach to speech dereverberation that involves joint estimation of the dry speech signal and of the room impulse response. We explore deep learning models that apply to each task separately, and how these can be combined in a joint model with shared parameters.

Motivation & Objective

• To address the challenge of single-channel speech dereverberation in non-ideal acoustic environments where reverberation degrades speech quality.
• To overcome limitations of traditional inverse filtering and masking-based methods that assume independence or short-duration effects.
• To improve performance by jointly estimating both the dry speech signal and the room impulse response (RIR) using shared deep learning parameters.
• To explore whether joint modeling can outperform separate estimation of speech and RIR, leveraging paired training data of dry speech, RIRs, and reverberant outputs.

Proposed method

• The method uses magnitude STFT representations of reverberant speech as input, with normalization applied per STFT frame.
• A U-Net architecture is employed for direct estimation of the dry speech signal, trained to predict the magnitude spectrogram from the reverberant input.
• A separate convolutional neural network (CNN) is designed to estimate the RIR, with a 1D convolutional structure applied across the time-frequency grid to predict the impulse response shape.
• The models are trained in a supervised manner using paired data of dry speech, RIRs, and corresponding reverberant signals generated via convolution.
• The joint training framework is designed to allow shared parameters between the two tasks, aiming to improve generalization and mutual performance.
• Future work includes incorporating phase information and using sequential models like LSTMs or CRNNs to better capture the temporal dynamics of RIRs.

Experimental results

Research questions

• RQ1 Can joint estimation of the dry speech signal and room impulse response (RIR) via deep learning outperform separate estimation of each component?
• RQ2 How does the inclusion of shared parameters in a joint model improve the accuracy of speech dereverberation compared to independent models?
• RQ3 Why do standard convolutional models fail to precisely estimate RIRs, which exhibit a sharp initial impulse followed by rapid decay?
• RQ4 To what extent can a U-Net-based approach for direct speech estimation be improved by incorporating phase information or sequential modeling?
• RQ5 Can time-domain or complex STFT representations enhance RIR estimation beyond magnitude-only STFT inputs?

Key findings

• The U-Net model achieved state-of-the-art performance for direct dry speech estimation from reverberant inputs, outperforming fully connected and Bi-GRU architectures.
• The Bi-GRU model introduced significant artifacts and was deemed unsuitable for practical use despite its sequential modeling capability.
• The convolutional model for RIR estimation produced outputs with correct decay envelopes but lacked precision in capturing the initial sharp impulse, indicating architectural limitations.
• The RIR estimation task is inherently difficult for standard CNNs due to the need to concentrate energy into a small time window, a structure not well-captured by typical DNN activations.
• The authors identify that phase information and sequential modeling (e.g., LSTMs or CRNNs) are key to improving RIR estimation, and plan to explore these in future work.
• Data preparation, including grouping similar RIRs and limiting group sizes, helped maintain balanced and representative training, validation, and test sets.