[Paper Review] Multichannel Speech Enhancement by Raw Waveform-mapping using Fully Convolutional Networks

Changle Liu, Szu‐Wei Fu+4 more|arXiv (Cornell University)|Sep 26, 2019

Speech and Audio Processing64 references4 citations

TL;DR

This paper proposes SDFCN and rSDFCN, fully convolutional networks with Sinc and dilated convolutions, for multichannel speech enhancement directly in the time domain. By bypassing spectral estimation, the method achieves superior denoising performance on dual-channel inner-ear and distributed microphone setups, with residual connections further improving results.

ABSTRACT

In recent years, waveform-mapping-based speech enhancement (SE) methods have garnered significant attention. These methods generally use a deep learning model to directly process and reconstruct speech waveforms. Because both the input and output are in waveform format, the waveform-mapping-based SE methods can overcome the distortion caused by imperfect phase estimation, which may be encountered in spectral-mapping-based SE systems. So far, most waveform-mapping-based SE methods have focused on single-channel tasks. In this paper, we propose a novel fully convolutional network (FCN) with Sinc and dilated convolutional layers (termed SDFCN) for multichannel SE that operates in the time domain. We also propose an extended version of SDFCN, called the residual SDFCN (termed rSDFCN). The proposed methods are evaluated on two multichannel SE tasks, namely the dual-channel inner-ear microphones SE task and the distributed microphones SE task. The experimental results confirm the outstanding denoising capability of the proposed SE systems on both tasks and the benefits of using the residual architecture on the overall SE performance.

Motivation & Objective

• To address the limitations of spectral-mapping-based speech enhancement, particularly phase estimation errors, by developing a time-domain waveform-mapping approach.
• To extend waveform-mapping techniques from single-channel to multichannel scenarios, enabling robust speech enhancement in complex acoustic environments.
• To design a fully convolutional network architecture with Sinc and dilated convolutions for efficient and effective multichannel speech enhancement.
• To evaluate the proposed method on two distinct multichannel speech enhancement tasks: inner-ear microphones and distributed microphones.
• To investigate the impact of residual connections on enhancing the performance and stability of the waveform-mapping model.

Proposed method

• The proposed SDFCN employs Sinc convolutional layers to learn optimal filter banks directly from raw waveforms, enabling parameter-efficient spectral modeling.
• Dilated convolutions are incorporated to expand the receptive field without increasing parameters, allowing the network to capture long-range temporal dependencies.
• The residual SDFCN (rSDFCN) integrates skip connections to improve gradient flow and training stability, enhancing feature learning in deep architectures.
• The model processes multichannel input waveforms directly in the time domain, avoiding phase estimation errors inherent in spectral-domain methods.
• A multihead attention mechanism is not used; instead, the method relies on the hierarchical feature learning of dilated and Sinc convolutions for robust enhancement.
• The network is trained end-to-end using a mean squared error loss between the enhanced and clean waveforms.

Experimental results

Research questions

• RQ1 Can a fully convolutional network with Sinc and dilated convolutions effectively perform multichannel speech enhancement in the time domain?
• RQ2 How does the proposed SDFCN architecture compare to existing waveform-mapping models in terms of denoising performance on multichannel data?
• RQ3 What is the impact of residual connections on the performance and convergence of the waveform-mapping model in multichannel settings?
• RQ4 Does the proposed method outperform conventional spectral-mapping-based systems in terms of speech quality and intelligibility on dual-channel inner-ear and distributed microphone tasks?
• RQ5 Can the Sinc and dilated convolution design reduce parameter count while maintaining or improving enhancement accuracy?

Key findings

• The proposed SDFCN achieves significant improvements in speech quality and noise reduction on both the dual-channel inner-ear microphone and distributed microphone tasks.
• The rSDFCN variant outperforms the base SDFCN, demonstrating that residual connections enhance model stability and performance.
• The method achieves competitive results compared to state-of-the-art spectral-mapping systems, particularly in preserving speech intelligibility and reducing artifacts.
• The use of Sinc convolutions enables the network to learn optimal filter responses directly from raw waveforms, improving modeling efficiency.
• The dilated convolutional layers effectively capture long-range temporal dependencies without increasing model complexity.
• The end-to-end training on raw waveforms eliminates the need for phase reconstruction, reducing distortion in enhanced outputs.