[Paper Review] Dual-Path Transformer Network: Direct Context-Aware Modeling for End-to-End Monaural Speech Separation

Jingjing Chen, Qirong Mao+1 more|arXiv (Cornell University)|Jul 28, 2020

Speech and Audio Processing32 references36 citations

TL;DR

Introduces a dual-path transformer network (DPTNet) that enables direct context-aware modeling for end-to-end monaural speech separation, achieving state-of-the-art results on WSJ0-2mix and strong performance on LS-2mix.

ABSTRACT

The dominant speech separation models are based on complex recurrent or convolution neural network that model speech sequences indirectly conditioning on context, such as passing information through many intermediate states in recurrent neural network, leading to suboptimal separation performance. In this paper, we propose a dual-path transformer network (DPTNet) for end-to-end speech separation, which introduces direct context-awareness in the modeling for speech sequences. By introduces a improved transformer, elements in speech sequences can interact directly, which enables DPTNet can model for the speech sequences with direct context-awareness. The improved transformer in our approach learns the order information of the speech sequences without positional encodings by incorporating a recurrent neural network into the original transformer. In addition, the structure of dual paths makes our model efficient for extremely long speech sequence modeling. Extensive experiments on benchmark datasets show that our approach outperforms the current state-of-the-arts (20.6 dB SDR on the public WSj0-2mix data corpus).

Motivation & Objective

• Motivate improving monaural speech separation by enabling direct context interaction rather than indirect, multi-stage context conditioning.
• Develop an end-to-end time-domain model that can handle extremely long speech sequences efficiently.
• Incorporate an improved transformer that learns order information without positional encodings by integrating a recurrent component.
• Leverage a dual-path architecture to model local and global dependencies in long sequences while remaining computationally feasible.

Proposed method

• Use an encoder–separator–decoder framework modeled after Conv-TasNet, with a 1-D convolutional encoder and a transposed convolutional decoder.
• Introduce a dual-path transformer processing stage consisting of segmentation, intra-transformer (local) and inter-transformer (global) blocks, and overlap-add.
• Replace the first fully connected layer in the transformer's feed-forward network with a recurrent neural network to learn sequence order without positional encodings.
• Employ a multi-head self-attention transformer within both intra- and inter-transformers to enable direct context interaction among speech sequence elements.
• Train with permutation invariant training (uPIT) to maximize scale-invariant SDR (SI-SNR) as the objective.

Experimental results

Research questions

• RQ1 Can direct context-aware modeling via a transformer improve end-to-end monaural speech separation compared to traditional RNN/CNN-based models?
• RQ2 Does a dual-path transformer structure enable efficient modeling of extremely long speech sequences while preserving separation quality?
• RQ3 Can ordering information be learned without positional encodings by integrating an RNN into the transformer's feed-forward network?
• RQ4 What are the comparative gains of DPTNet on standard benchmarks (WSJ0-2mix and LS-2mix) relative to state-of-the-art methods?

Key findings

• DPTNet achieves 20.2 dB SI-SNR and 20.6 dB SDR on WSJ0-2mix with a model size of 2.69M, outperforming previous state-of-the-art.
• On LS-2mix, DPTNet attains 16.2 dB SI-SNR and 16.8 dB SDR, surpassing baselines DSP-TasNet and DPRNN.
• The model demonstrates strong generalization across datasets, maintaining superiority over baselines that include Conv-TasNet and DPRNN.
• The dual-path design enables efficient long-sequence modeling, balancing direct context interaction with computational practicality.