[论文解读] All Information is Necessary: Integrating Speech Positive and Negative Information by Contrastive Learning for Speech Enhancement

Xinmeng Xu, Weiping Tu+2 more|arXiv (Cornell University)|Apr 26, 2023

Speech and Audio Processing被引用 4

一句话总结

该论文提出CMCR-Net，一种单耳语音增强模型，通过对比学习整合语音正向（相关）与负向（无关）特征。通过引入具有对比注意力和交互注意力的协作模块，以及对比正则化损失，该模型在计算成本更低的前提下，提升了语音质量和可懂度，实现了SOTA性能。

ABSTRACT

Monaural speech enhancement (SE) is an ill-posed problem due to the irreversible degradation process. Recent methods to achieve SE tasks rely solely on positive information, e.g., ground-truth speech and speech-relevant features. Different from the above, we observe that the negative information, such as original speech mixture and speech-irrelevant features, are valuable to guide the SE model training procedure. In this study, we propose a SE model that integrates both speech positive and negative information for improving SE performance by adopting contrastive learning, in which two innovations have consisted. (1) We design a collaboration module (CM), which contains two parts, contrastive attention for separating relevant and irrelevant features via contrastive learning and interactive attention for establishing the correlation between both speech features in a learnable and self-adaptive manner. (2) We propose a contrastive regularization (CR) built upon contrastive learning to ensure that the estimated speech is pulled closer to the clean speech and pushed far away from the noisy speech in the representation space by integrating self-supervised models. We term the proposed SE network with CM and CR as CMCR-Net. Experimental results demonstrate that our CMCR-Net achieves comparable and superior performance to recent approaches.

研究动机与目标

• 解决现有语音增强模型仅依赖正向信息（如干净语音）而忽略语音无关特征的局限性。
• 克服双分支模型分别建模语音与噪声所导致的高参数量与训练难度。
• 通过显式建模语音相关与无关特征之间的相关性，提升判别性表征学习能力。
• 开发一种模型无关的正则化技术，在不增加推理复杂度的前提下提升泛化能力。

提出的方法

• 设计一个协作模块（CM），利用对比注意力在表征空间中通过自注意力和对比学习分离语音相关与无关特征。
• 在CM中引入交互注意力，以自适应方式动态学习并融合相关与无关特征之间的相关性。
• 提出基于对比学习的对比正则化（CR），在嵌入空间中使增强后的语音表征更接近干净语音，远离噪声语音。
• 将CM与CR集成到编码器-解码器架构中，形成CMCR-Net，实现端到端训练且额外参数极少。
• 使用UniSpeech-SAT作为预训练特征编码器，以提升表征质量，并确保与SOTA模型的公平比较。
• 采用重建损失与对比正则化联合训练模型，并在推理阶段移除CR以避免计算开销。

Figure 1: The architecture of the proposed CMCR-Net. It adopts encoder-decoder architecture as the main body. In addition, the loss function of the proposed CMCR-Net consists of MSE loss and contrastive regularization.

实验结果

研究问题

• RQ1 与仅依赖正向信息的模型相比，整合语音无关（负向）特征是否能提升单耳语音增强性能？
• RQ2 单分支架构如何在不单独建模噪声的情况下，有效建模语音相关与无关特征之间的相关性？
• RQ3 对比学习是否能有效应用于语音增强中，通过在嵌入空间中拉近干净语音与拉远噪声语音，提升判别性表征学习？
• RQ4 所提出的对比正则化（CR）是否能在不增加推理成本的前提下，提升模型在多种噪声类型与信噪比条件下的泛化能力？
• RQ5 CMCR-Net框架是否能有效迁移至Voice Bank + DEMAND与AVSpeech + AudioSet等多样化数据集，并超越现有SOTA模型？

主要发现

• 在AVSpeech + AudioSet数据集上，CMCR-Net的PESQ得分为3.26，STOI为90.53%，优于所有对比的SOTA方法，包括DCCRN、PHASEN、TFT-Net、SA-TCN和U-Former。
• 在Voice Bank + DEMAND数据集上，CMCR-Net达到最高PESQ（3.10）与STOI（93.04%），模型参数量最小（2.45M），推理时间最短（每秒音频0.42秒）。
• 对比正则化（CR）在应用于DCCRN时，STOI最高提升1.60个百分点，PESQ提升0.06，证明其具有模型无关的有效性。
• 消融实验表明，移除CR后STOI下降1.78个百分点，PESQ下降0.08，证实CR对性能有显著贡献。
• 可视化结果表明，CMCR-Net生成的频谱图更干净，噪声残留更少，表明其具有更好的噪声抑制能力。
• 该模型在多种噪声类型（如babble、factory、cafeteria）下泛化良好，即使在三种噪声类型组合时性能仅略有下降，STOI与PESQ保持较高水平。

Figure 2: The visualization of attention maps obtained by softmax and inner product in self-attention and contrastive attention, respectively. We can see that self-attention pays attention to speech-irrelevant features and ignore several speech-relevant features, while our contrastive attention can