[Paper Review] Real-time speech enhancement using equilibriated RNN

Daiki Takeuchi, Kohei Yatabe+3 more|arXiv (Cornell University)|Feb 14, 2020

Speech and Audio Processing27 references4 citations

TL;DR

This paper proposes a real-time speech enhancement method using a causal equilibriated recurrent neural network (ERNN) to achieve performance comparable to LSTM with significantly fewer parameters. By leveraging iterative weight sharing to stabilize gradients without gated units, ERNN enables efficient, low-complexity inference suitable for resource-limited devices while maintaining high-quality speech enhancement.

ABSTRACT

We propose a speech enhancement method using a causal deep neural network~(DNN) for real-time applications. DNN has been widely used for estimating a time-frequency~(T-F) mask which enhances a speech signal. One popular DNN structure for that is a recurrent neural network~(RNN) owing to its capability of effectively modelling time-sequential data like speech. In particular, the long short-term memory (LSTM) is often used to alleviate the vanishing/exploding gradient problem which makes the training of an RNN difficult. However, the number of parameters of LSTM is increased as the price of mitigating the difficulty of training, which requires more computational resources. For real-time speech enhancement, it is preferable to use a smaller network without losing the performance. In this paper, we propose to use the equilibriated recurrent neural network~(ERNN) for avoiding the vanishing/exploding gradient problem without increasing the number of parameters. The proposed structure is causal, which requires only the information from the past, in order to apply it in real-time. Compared to the uni- and bi-directional LSTM networks, the proposed method achieved the similar performance with much fewer parameters.

Motivation & Objective

• Address the challenge of deploying deep learning-based speech enhancement in real-time applications with limited computational resources.
• Overcome the vanishing/exploding gradient problem in recurrent networks without increasing model complexity.
• Develop a causal RNN architecture that maintains long-term temporal dependencies while minimizing parameter count.
• Enable efficient inference on edge devices by reducing model size without sacrificing enhancement quality.
• Demonstrate that ERNN can outperform pruned LSTM models in both performance and efficiency trade-offs.

Proposed method

• Use the equilibriated recurrent neural network (ERNN), which applies the same fully connected layer iteratively to the hidden state to stabilize gradients.
• Ensure causality by only using past information, making the network suitable for real-time processing.
• Implement the ERNN as a T-F mask estimator in the STFT domain, where the mask is applied to noisy spectrograms to recover enhanced speech.
• Train the network via back-propagation with gradient stability guaranteed by the iterative weight-sharing mechanism.
• Control model capacity and computational cost by adjusting the number of iterations $K$ and the hidden dimension $N_{\rm h}$.
• Use a fully connected DNN with ReLU activation as the base module $\mathscr{F}$ in the ERNN architecture.

Experimental results

Research questions

• RQ1 Can ERNN achieve comparable speech enhancement performance to LSTM while using significantly fewer parameters?
• RQ2 Does the ERNN architecture effectively mitigate the vanishing/exploding gradient problem without gated units?
• RQ3 How does the number of iterations $K$ and hidden dimension $N_{\rm h}$ affect the trade-off between performance and computational cost?
• RQ4 Can ERNN maintain high-quality enhancement in real-time applications despite its reduced parameter count?
• RQ5 How does the performance of ERNN compare to both causal LSTM and non-causal BLSTM in terms of objective metrics and parameter efficiency?

Key findings

• The proposed ERNN-based method achieved PESQ scores of 2.54 and 3.74 on the 512-point STFT, matching or exceeding those of LSTM with 1/14th the parameters.
• With only 231k parameters, ERNN achieved a PESQ of 2.40 and CSIG of 3.56, comparable to a 3.81M-parameter LSTM.
• ERNN outperformed a pruned LSTM model with similar parameter count, showing better performance-complexity trade-offs.
• The method achieved performance close to non-causal BLSTM (which cannot be used in real time) with less than 1/36th the parameters.
• Reducing the number of iterations $K$ from 5 to 1 had minimal impact on performance, enabling further computational savings.
• The ERNN model maintained stable gradients without gated units, enabling effective learning of long-term dependencies despite low parameter count.