[论文解读] Monaural Speech Enhancement Using a Multi-Branch Temporal Convolutional Network

Qiquan Zhang, Aaron Nicolson+3 more|arXiv (Cornell University)|Dec 27, 2019

Speech and Audio Processing被引用 6

一句话总结

该论文提出了一种用于单耳语音增强的多分支时序卷积网络（MB-TCN），通过使用因果空洞卷积和残差学习的拆分-转换-聚合设计，高效捕捉长期时序依赖关系。MB-TCN在五项客观指标上优于最先进方法（包括ResLSTM、TCN和基于GAN的模型），同时保持了高参数效率。

ABSTRACT

Deep learning has achieved substantial improvement on single-channel speech enhancement tasks. However, the performance of multi-layer perceptions (MLPs)-based methods is limited by the ability to capture the long-term effective history information. The recurrent neural networks (RNNs), e.g., long short-term memory (LSTM) model, are able to capture the long-term temporal dependencies, but come with the issues of the high latency and the complexity of training.To address these issues, the temporal convolutional network (TCN) was proposed to replace the RNNs in various sequence modeling tasks. In this paper we propose a novel TCN model that employs multi-branch structure, called multi-branch TCN (MB-TCN), for monaural speech enhancement.The MB-TCN exploits split-transform-aggregate design, which is expected to obtain strong representational power at a low computational complexity.Inspired by the TCN, the MB-TCN model incorporates one dimensional causal dilated CNN and residual learning to expand receptive fields for capturing long-term temporal contextual information.Our extensive experimental investigation suggests that the MB-TCNs outperform the residual long short-term memory networks (ResLSTMs), temporal convolutional networks (TCNs), and the CNN networks that employ dense aggregations in terms of speech intelligibility and quality, while providing superior parameter efficiency. Furthermore, our experimental results demonstrate that our proposed MB-TCN model is able to outperform multiple state-of-the-art deep learning-based speech enhancement methods in terms of five widely used objective metrics.

研究动机与目标

• 解决MLP和RNN在捕捉单耳语音增强中长期语音依赖关系方面的局限性。
• 克服LSTM等RNN在序列建模任务中高延迟和高训练复杂度的问题。
• 在保持低计算复杂度和参数效率的同时，提升语音质量和可懂度。
• 开发一种深度学习模型，以最小的架构开销有效捕捉长程时序上下文。
• 在客观指标和模型效率方面，全面超越现有最先进方法。

提出的方法

• MB-TCN采用拆分-转换-聚合架构，在计算成本较低的前提下增强表征能力。
• 使用一维因果空洞卷积扩展感受野，以建模长程时序依赖关系。
• 引入残差连接以稳定训练并支持更深的网络结构。
• 多分支设计允许并行处理不同空洞卷积核尺度，提升特征表示能力。
• 该模型被集成到Deep Xi框架中，用于估计先验信噪比（SNR），以支持SRWF等统计增强方法。
• 架构设计注重参数效率，超参数可调，以平衡性能与模型大小。

Figure 1 : An example of 1-D dilated causal convolution with the kernel size $k=3$ and the dilation rate $d=1,2,4$ . This forms a large receptive field over the input.

实验结果

研究问题

• RQ1 多分支TCN架构是否能在单耳语音增强任务中超越标准TCN和ResLSTM模型？
• RQ2 结合空洞卷积和残差学习的拆分-转换-聚合设计，是否能比RNN或标准CNN实现更优的长期上下文建模？
• RQ3 与最先进方法相比，MB-TCN在客观语音质量与可懂度指标上的提升程度如何？
• RQ4 与基于GAN和密集连接的模型相比，MB-TCN在参数效率和计算成本方面表现如何？
• RQ5 MB-TCN是否能以更少的参数量实现优于现有SOTA模型的性能？

主要发现

• 在CSIG、CBAK和COVL指标上，MB-TCN分别较DFL提升0.35、0.08和0.37。
• 在CSIG、CBAK、COVL和PESQ指标上，分别较MetricGAN提升0.22、0.23、0.17和0.08。
• 相较于MMSE-GAN这一强基准模型，MB-TCN实现0.41的PESQ提升和约1%的STOI增益。
• 尽管参数量多于MMSE-GAN，MB-TCN展现出显著更优的性能，并避免了GAN训练的不稳定性。
• MB-TCN在噪声抑制方面表现更优，语音失真更少，如频谱图对比所示。
• 该模型在性能与参数效率之间实现了最佳平衡，优于Wave-U-Net、SEGAN和MDPhD等模型，在指标和模型尺寸上均表现更优。

Figure 3 : Proposed multi-branch TCN (MB-TCN) architecture. It is composed of a fully-connected first layer, FC , followed by N multi-branch TCN blocks, and then a fully-connected output layer, O that employs sigmoid units. $\copyright$ represent the concatenation operation. The kernel size, output