[Paper Review] All Information is Necessary: Integrating Speech Positive and Negative Information by Contrastive Learning for Speech Enhancement

Xinmeng Xu, Weiping Tu+2 more|arXiv (Cornell University)|Apr 26, 2023

Speech and Audio Processing4 citations

TL;DR

This paper proposes CMCR-Net, a monaural speech enhancement model that integrates both speech-positive (relevant) and speech-negative (irrelevant) features via contrastive learning. By introducing a collaboration module with contrastive and interactive attention, and a contrastive regularization loss, the model improves speech quality and intelligibility, achieving state-of-the-art performance with lower computational cost.

ABSTRACT

Monaural speech enhancement (SE) is an ill-posed problem due to the irreversible degradation process. Recent methods to achieve SE tasks rely solely on positive information, e.g., ground-truth speech and speech-relevant features. Different from the above, we observe that the negative information, such as original speech mixture and speech-irrelevant features, are valuable to guide the SE model training procedure. In this study, we propose a SE model that integrates both speech positive and negative information for improving SE performance by adopting contrastive learning, in which two innovations have consisted. (1) We design a collaboration module (CM), which contains two parts, contrastive attention for separating relevant and irrelevant features via contrastive learning and interactive attention for establishing the correlation between both speech features in a learnable and self-adaptive manner. (2) We propose a contrastive regularization (CR) built upon contrastive learning to ensure that the estimated speech is pulled closer to the clean speech and pushed far away from the noisy speech in the representation space by integrating self-supervised models. We term the proposed SE network with CM and CR as CMCR-Net. Experimental results demonstrate that our CMCR-Net achieves comparable and superior performance to recent approaches.

Motivation & Objective

• Address the limitation of existing speech enhancement models that rely only on positive information (e.g., clean speech) and ignore speech-irrelevant features.
• Overcome the high parameter cost and training difficulty of two-branch models that separately model speech and noise.
• Improve discriminative representation learning by explicitly modeling correlations between speech-relevant and speech-irrelevant features.
• Develop a model-agnostic regularization technique that enhances generalization without increasing inference complexity.

Proposed method

• Design a collaboration module (CM) that uses contrastive attention to separate speech-relevant and irrelevant features in the representation space using self-attention and contrastive learning.
• Introduce interactive attention within CM to dynamically learn and fuse correlations between relevant and irrelevant features in a self-adaptive manner.
• Propose contrastive regularization (CR) based on contrastive learning to pull enhanced speech representations closer to clean speech and farther from noisy speech in the embedding space.
• Integrate CM and CR into an encoder-decoder architecture to form CMCR-Net, enabling end-to-end training with minimal additional parameters.
• Use UniSpeech-SAT as a pre-trained feature encoder to improve representation quality and ensure fair comparison with SOTA models.
• Train the model using a combination of reconstruction loss and contrastive regularization, with CR removed during inference to avoid computational overhead.

Figure 1: The architecture of the proposed CMCR-Net. It adopts encoder-decoder architecture as the main body. In addition, the loss function of the proposed CMCR-Net consists of MSE loss and contrastive regularization.

Experimental results

Research questions

• RQ1 Can integrating speech-irrelevant (negative) features improve monaural speech enhancement performance compared to models relying only on positive information?
• RQ2 How can a single-branch architecture effectively model the correlation between speech-relevant and speech-irrelevant features without separate modeling of noise?
• RQ3 Can contrastive learning be effectively applied to enhance discriminative representation learning in speech enhancement by pulling clean speech closer and noisy speech farther in the embedding space?
• RQ4 Does the proposed contrastive regularization (CR) improve model generalization across diverse noise types and SNR conditions without increasing inference cost?
• RQ5 Can the CMCR-Net framework be effectively transferred to and outperform existing SOTA models on diverse datasets like Voice Bank + DEMAND and AVSpeech + AudioSet?

Key findings

• CMCR-Net achieves a PESQ score of 3.26 and STOI of 90.53% on the AVSpeech + AudioSet dataset, outperforming all compared SOTA methods including DCCRN, PHASEN, TFT-Net, SA-TCN, and U-Former.
• On the Voice Bank + DEMAND dataset, CMCR-Net achieves the highest PESQ (3.10) and STOI (93.04%), with the lowest model size (2.45M parameters) and inference time (0.42s per second of audio).
• The contrastive regularization (CR) improves STOI by up to 1.60 percentage points and PESQ by 0.06 when applied to DCCRN, demonstrating its model-agnostic effectiveness.
• Ablation studies show that removing CR reduces STOI by 1.78 percentage points and PESQ by 0.08, confirming CR’s significant contribution to performance.
• Visualization shows that CMCR-Net produces cleaner spectrograms with reduced noise residue compared to baseline models, indicating better noise suppression.
• The model generalizes well across multiple noise types (babble, factory, cafeteria), maintaining high performance even when three noise types are combined, with only a modest drop in STOI and PESQ as noise complexity increases.

Figure 2: The visualization of attention maps obtained by softmax and inner product in self-attention and contrastive attention, respectively. We can see that self-attention pays attention to speech-irrelevant features and ignore several speech-relevant features, while our contrastive attention can