[Paper Review] Interactive Speech and Noise Modeling for Speech Enhancement

Chengyu Zheng, Xiulian Peng+3 more|arXiv (Cornell University)|Dec 17, 2020

Speech and Audio Processing4 citations

TL;DR

This paper proposes SN-Net, a two-branch convolutional neural network that simultaneously models speech and noise with cross-branch interaction modules to enhance speech quality and suppress residual noise. By integrating residual-convolution-and-attention (RA) blocks and enabling feature exchange between speech and noise branches, SN-Net achieves state-of-the-art performance on both speech enhancement and speaker separation tasks, outperforming prior methods by large margins on public benchmarks like Voice Bank + DEMAND and DNS Challenge.

ABSTRACT

Speech enhancement is challenging because of the diversity of background noise types. Most of the existing methods are focused on modelling the speech rather than the noise. In this paper, we propose a novel idea to model speech and noise simultaneously in a two-branch convolutional neural network, namely SN-Net. In SN-Net, the two branches predict speech and noise, respectively. Instead of information fusion only at the final output layer, interaction modules are introduced at several intermediate feature domains between the two branches to benefit each other. Such an interaction can leverage features learned from one branch to counteract the undesired part and restore the missing component of the other and thus enhance their discrimination capabilities. We also design a feature extraction module, namely residual-convolution-and-attention (RA), to capture the correlations along temporal and frequency dimensions for both the speech and the noises. Evaluations on public datasets show that the interaction module plays a key role in simultaneous modeling and the SN-Net outperforms the state-of-the-art by a large margin on various evaluation metrics. The proposed SN-Net also shows superior performance for speaker separation.

Motivation & Objective

• To address the limitation of existing speech enhancement methods that focus only on speech modeling and ignore noise characteristics.
• To improve speech enhancement by modeling both speech and noise simultaneously with bidirectional information flow between branches.
• To reduce residual noise and speech distortion by leveraging complementary features from speech and noise branches through interaction modules.
• To develop a robust feature extraction mechanism capable of capturing global temporal and frequency dependencies in both speech and noise.
• To extend the proposed framework to speaker separation, demonstrating its generalizability beyond single-speaker enhancement.

Proposed method

• SN-Net employs a two-branch encoder-decoder architecture, with shared structure for speech and noise branches, enabling joint modeling of both signals.
• Interaction modules are inserted at multiple intermediate layers to allow feature exchange between speech and noise branches, enhancing discrimination and feature refinement.
• The residual-convolution-and-attention (RA) block combines residual connections, depthwise separable convolutions, and parallel temporal and frequency-wise self-attention to model long-range dependencies.
• Temporal self-attention captures global patterns across time frames, while frequency-wise self-attention models spectral correlations across frequency bins.
• The network is trained using a combination of reconstruction loss and perceptual metrics, with permutation-invariant training applied for speaker separation.
• Feature maps from both branches are processed through RA blocks, which progressively refine representations before final output generation.

Experimental results

Research questions

• RQ1 Can simultaneous modeling of speech and noise with cross-branch interaction improve speech enhancement performance compared to unidirectional speech-only modeling?
• RQ2 How does the integration of self-attention mechanisms across time and frequency dimensions affect feature representation in noisy speech?
• RQ3 To what extent does the interaction between speech and noise branches reduce residual noise and speech distortion?
• RQ4 Can the proposed two-branch framework be effectively extended to speaker separation, and how does it compare to existing SOTA methods?
• RQ5 What is the contribution of the RA block and interaction modules in improving overall enhancement quality and robustness across diverse noise types?

Key findings

• SN-Net achieves a PESQ score of 3.12 on Voice Bank + DEMAND, outperforming all compared SOTA methods, including GAN-based and self-attention models.
• On the DNS Challenge dataset, SN-Net achieves an SDRi of 8.39 dB and PESQ of 2.50, surpassing Conv-TasNet (SDRi: 7.57 dB, PESQ: 2.14) by 0.82 dB and 0.36 points, respectively.
• The interaction module contributes significantly to performance, with ablation studies confirming its key role in reducing residual noise and improving speech clarity.
• The RA block effectively captures long-range temporal and frequency dependencies, as evidenced by attention maps showing global focus in speech and localized patterns in noise.
• In speaker separation, SN-Net improves SDRi by 0.82 dB and PESQ by 0.36 points over Conv-TasNet, demonstrating strong generalization to multi-source separation.
• SN-Net achieves the highest scores across all metrics on both datasets, indicating superior robustness and enhancement quality under diverse noise conditions.