[Paper Review] Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separation

Yi Luo, Zhuo Chen+1 more|arXiv (Cornell University)|Oct 14, 2019

Speech and Audio ProcessingComputer Science36 references50 citations

TL;DR

Introduces Dual-path RNN (DPRNN) to model extremely long sequences by alternating intra-chunk and inter-chunk RNNs, replacing 1-D CNNs in TasNet to achieve state-of-the-art WSJ0-2mix with a much smaller model.

ABSTRACT

Recent studies in deep learning-based speech separation have proven the superiority of time-domain approaches to conventional time-frequency-based methods. Unlike the time-frequency domain approaches, the time-domain separation systems often receive input sequences consisting of a huge number of time steps, which introduces challenges for modeling extremely long sequences. Conventional recurrent neural networks (RNNs) are not effective for modeling such long sequences due to optimization difficulties, while one-dimensional convolutional neural networks (1-D CNNs) cannot perform utterance-level sequence modeling when its receptive field is smaller than the sequence length. In this paper, we propose dual-path recurrent neural network (DPRNN), a simple yet effective method for organizing RNN layers in a deep structure to model extremely long sequences. DPRNN splits the long sequential input into smaller chunks and applies intra- and inter-chunk operations iteratively, where the input length can be made proportional to the square root of the original sequence length in each operation. Experiments show that by replacing 1-D CNN with DPRNN and apply sample-level modeling in the time-domain audio separation network (TasNet), a new state-of-the-art performance on WSJ0-2mix is achieved with a 20 times smaller model than the previous best system.

Motivation & Objective

• Motivate the need for effective long-sequence modeling in time-domain speech separation.
• Propose a simple architecture that splits long sequences into chunks and applies local and global RNNs alternately.
• Demonstrate that sublinear input length (O(sqrt(L))) improves optimization while maintaining performance.
• Show that replacing CNNs with DPRNN in TasNet achieves state-of-the-art results with smaller models.

Proposed method

• Segment long input into overlapped chunks to form a 3-D tensor.
• Apply intra-chunk bi-directional RNN to each chunk independently (local modeling).
• Apply inter-chunk RNN across chunks to capture global dependencies (utterance-level modeling).
• Use a residual connection and layer normalization after each DPRNN block.
• Stack multiple DPRNN blocks with overlap-add to reconstruct the sequence.

Experimental results

Research questions

• RQ1 Can a dual-path RNN architecture effectively model extremely long sequences for time-domain speech separation?
• RQ2 Does alternating intra-chunk and inter-chunk processing provide gains over fixed-receptive-field CNN baselines?
• RQ3 What is the impact of DPRNN on model size and performance on WSJ0-2mix and noisy reverberant conditions?

Key findings

• DPRNN-TasNet improves SI-SNRi by 4.6% over TCN-TasNet with 49% smaller model size on WSJ0-2mix.
• Replacing 1-D CNN with DPRNN enables smaller models to achieve comparable or better performance, with best SI-SNRi achieved when encoder window is 2 samples and frames exceed 30k.
• DPRNN-TasNet establishes a new state-of-the-art SI-SNRi/SDRi on WSJ0-2mix with a 20x smaller model than FurcaNeXt.
• DPRNN shows strong performance gains under noisy reverberant conditions, with higher SI-SNRi and lower WER compared to TCN-based models.