[Paper Review] Speech Enhancement Using Multi-Stage Self-Attentive Temporal Convolutional Networks

Ju Lin, A. van Wijngaarden+2 more|arXiv (Cornell University)|Feb 24, 2021

Speech and Audio Processing53 references4 citations

TL;DR

This paper proposes a multi-stage self-attentive temporal convolutional network (SA-TCN) for speech enhancement, where each stage refines the output of the previous one using self-attention and dilated TCN blocks with increasing receptive fields. The method achieves state-of-the-art performance on LibriSpeech and VCTK datasets with a 5-stage model, improving PESQ, STOI, and SI-SDR scores while maintaining low parameter count and computational efficiency.

ABSTRACT

Multi-stage learning is an effective technique to invoke multiple deep-learning modules sequentially. This paper applies multi-stage learning to speech enhancement by using a multi-stage structure, where each stage comprises a self-attention (SA) block followed by stacks of temporal convolutional network (TCN) blocks with doubling dilation factors. Each stage generates a prediction that is refined in a subsequent stage. A fusion block is inserted at the input of later stages to re-inject original information. The resulting multi-stage speech enhancement system, in short, multi-stage SA-TCN, is compared with state-of-the-art deep-learning speech enhancement methods using the LibriSpeech and VCTK data sets. The multi-stage SA-TCN system's hyper-parameters are fine-tuned, and the impact of the SA block, the fusion block and the number of stages are determined. The use of a multi-stage SA-TCN system as a front-end for automatic speech recognition systems is investigated as well. It is shown that the multi-stage SA-TCN systems perform well relative to other state-of-the-art systems in terms of speech enhancement and speech recognition scores.

Motivation & Objective

• To improve speech enhancement performance by leveraging multi-stage learning with self-attention and temporal convolutional networks.
• To address the degradation of speech information in deep sequential models by introducing a fusion block in later stages.
• To design a parameter-efficient architecture that maintains high performance across diverse noise conditions.
• To evaluate the system's effectiveness as a front-end for automatic speech recognition (ASR) systems.
• To investigate the impact of self-attention, fusion blocks, and the number of stages on overall enhancement quality.

Proposed method

• Each stage consists of a self-attention (SA) block followed by R stacks of L temporal convolutional network (TCN) blocks with doubling dilation factors.
• The TCN blocks use dilated 1D convolutions to expand the temporal receptive field without increasing parameters significantly.
• A fusion block is introduced from the third stage onward to re-inject original noisy features and preserve speech content.
• The model predicts a soft mask at each stage, implicitly approximating an ideal ratio mask (IRM), which is used to refine the enhanced signal iteratively.
• The system uses magnitude spectrograms as input and reconstructs the time-domain waveform using the original noisy phase.
• Hyper-parameters, including the number of stages, SA block usage, and fusion block integration, are fine-tuned using LibriSpeech and VCTK datasets.

Experimental results

Research questions

• RQ1 How does multi-stage learning with SA-TCN improve speech enhancement performance compared to single-stage models?
• RQ2 What is the impact of incorporating self-attention and fusion blocks on enhancement quality and parameter efficiency?
• RQ3 How does the number of stages affect performance, and does performance saturate beyond a certain number of stages?
• RQ4 Can the proposed multi-stage SA-TCN system serve as an effective front-end for automatic speech recognition (ASR) systems?
• RQ5 How does the model compare to state-of-the-art methods in terms of objective metrics like PESQ, STOI, and SI-SDR?

Key findings

• The 5-stage SA-TCN system achieves the best overall performance with a PESQ of 3.02, STOI of 0.9439, and SI-SDR of 18.48 on the VCTK dataset.
• Adding more stages consistently improves performance, though the marginal gain diminishes beyond four stages, indicating convergence toward an implicit upper bound.
• The inclusion of self-attention and fusion blocks leads to measurable improvements in PESQ, STOI, and CSIG scores across both LibriSpeech and VCTK datasets.
• The 5-stage SA-TCN model achieves state-of-the-art results with only 9.91 million parameters, demonstrating high efficiency and low computational cost.
• The system significantly enhances speech quality and intelligibility, as confirmed by spectrogram visualizations showing effective noise suppression while preserving speech patterns.
• When used as a front-end for ASR, the multi-stage SA-TCN system improves recognition scores, confirming its robustness and utility in downstream tasks.