[Paper Review] Dynamic Attention Based Generative Adversarial Network with Phase Post-Processing for Speech Enhancement

Andong Li, Chengshi Zheng+3 more|arXiv (Cornell University)|Jun 13, 2020

Speech and Audio Processing19 references4 citations

TL;DR

This paper proposes DARGAN, a dynamic attention-based generative adversarial network with recursive learning and phase post-processing for speech enhancement in the time-frequency domain. By iteratively refining noisy features through a recursive generator, applying dynamic attention to adaptively reweight features, and using a deep Griffin-Lim algorithm for phase reconstruction, DARGAN achieves state-of-the-art performance on the Voice Bank corpus, outperforming prior GAN- and non-GAN-based models in PESQ, CSIG, CBAK, and COVL metrics.

ABSTRACT

The generative adversarial networks (GANs) have facilitated the development of speech enhancement recently. Nevertheless, the performance advantage is still limited when compared with state-of-the-art models. In this paper, we propose a powerful Dynamic Attention Recursive GAN called DARGAN for noise reduction in the time-frequency domain. Different from previous works, we have several innovations. First, recursive learning, an iterative training protocol, is used in the generator, which consists of multiple steps. By reusing the network in each step, the noise components are progressively reduced in a step-wise manner. Second, the dynamic attention mechanism is deployed, which helps to re-adjust the feature distribution in the noise reduction module. Third, we exploit the deep Griffin-Lim algorithm as the module for phase postprocessing, which facilitates further improvement in speech quality. Experimental results on Voice Bank corpus show that the proposed GAN achieves state-of-the-art performance than previous GAN- and non-GAN-based models

Motivation & Objective

• To address the limitations of existing GAN-based speech enhancement models, particularly in phase estimation and feature representation.
• To improve speech quality and intelligibility in low SNR and unstable noise environments through a novel deep learning architecture.
• To overcome the phase mismatch problem common in magnitude-only spectral estimation by introducing a dedicated phase post-processing step.
• To enhance feature learning and noise suppression through recursive training and dynamic attention mechanisms.
• To achieve state-of-the-art performance in objective metrics across GAN-based and non-GAN-based speech enhancement models.

Proposed method

• The generator employs a recursive learning protocol, decomposing the noise reduction process into multiple stages where each stage refines the output from the previous one using a memory mechanism.
• A dynamic attention mechanism is integrated into the noise reduction module to adaptively reweight feature maps, improving representation learning and noise suppression.
• The model uses least-squares GAN (LS-GAN) with conditional GAN (cGAN) for training, where the generator maps noisy spectrograms to enhanced ones.
• Phase post-processing is performed using a deep Griffin-Lim algorithm (DGLA), which iteratively refines the phase by unfolding the optimization block multiple times.
• The training objective combines adversarial loss and L1-regularization loss, with hyperparameter λG controlling the trade-off between perceptual quality and reconstruction fidelity.
• The discriminator is trained to distinguish between real clean spectrograms and generated enhanced ones, enabling the generator to produce more realistic outputs.

Experimental results

Research questions

• RQ1 Can recursive learning in the generator improve speech enhancement performance by enabling iterative refinement of noisy features?
• RQ2 Does the integration of a dynamic attention mechanism lead to better feature representation and noise suppression in the time-frequency domain?
• RQ3 Can phase post-processing via deep Griffin-Lim significantly improve speech quality by correcting phase mismatches from magnitude-only estimation?
• RQ4 How does the proposed DARGAN model compare to state-of-the-art GAN-based and non-GAN-based speech enhancement models in objective metrics?
• RQ5 What is the optimal trade-off between adversarial and L1-regularization losses in the training objective for maximizing speech quality?

Key findings

• DARGAN achieves a PESQ score of 2.93 and CSIG of 4.30 with λG = 1, outperforming SEGAN by 0.80 in PESQ and 0.81 in CSIG.
• The model surpasses MetricGAN, a recent SOTA non-GAN model, with a 0.07 increase in PESQ and 0.08 in CSIG.
• Phase post-processing (PPP) with DGLA improves PESQ by 0.03 and CBAK by 0.02, demonstrating its effectiveness in phase refinement.
• The best performance is achieved at λG = 1, where further increases degrade performance due to excessive emphasis on L1 loss.
• DARGAN outperforms STFT-TCN, a DNS-Challenge ranked model, in all four metrics, confirming its superiority in real-world noisy conditions.
• Visual analysis confirms that DARGAN suppresses residual noise more effectively than SEGAN while preserving formant structure and speech details.