[Paper Review] Time-domain Speech Enhancement with Generative Adversarial Learning

Feiyang Xiao, Jian Guan+2 more|arXiv (Cornell University)|Mar 30, 2021

Speech and Audio Processing29 references4 citations

TL;DR

This paper proposes TSEGAN, a time-domain speech enhancement framework that integrates metric evaluation into a generative adversarial network to stabilize training and mitigate scale ambiguity caused by SI-SNR loss. By using a discriminator that evaluates speech quality via SI-SNR, TSEGAN achieves superior objective performance and better generalization than WGAN-based methods, with theoretical analysis explaining why Metric GAN outperforms WGAN.

ABSTRACT

Speech enhancement aims to obtain speech signals with high intelligibility and quality from noisy speech. Recent work has demonstrated the excellent performance of time-domain deep learning methods, such as Conv-TasNet. However, these methods can be degraded by the arbitrary scales of the waveform induced by the scale-invariant signal-to-noise ratio (SI-SNR) loss. This paper proposes a new framework called Time-domain Speech Enhancement Generative Adversarial Network (TSEGAN), which is an extension of the generative adversarial network (GAN) in time-domain with metric evaluation to mitigate the scaling problem, and provide model training stability, thus achieving performance improvement. In addition, we provide a new method based on objective function mapping for the theoretical analysis of the performance of Metric GAN, and explain why it is better than the Wasserstein GAN. Experiments conducted demonstrate the effectiveness of our proposed method, and illustrate the advantage of Metric GAN.

Motivation & Objective

• To address the instability in training caused by scale-invariant signal-to-noise ratio (SI-SNR) loss in time-domain speech enhancement models.
• To improve the generalization and objective performance of speech enhancement models by incorporating metric evaluation into the GAN framework.
• To provide a theoretical explanation for why Metric GAN outperforms WGAN in speech enhancement tasks.
• To develop a flexible framework that supports various TasNet-based generators for time-domain speech enhancement.

Proposed method

• The framework uses a TasNet-based generator to learn latent representations of noisy speech in the time domain.
• A metric evaluation discriminator is introduced to estimate the SI-SNR between enhanced and clean speech, replacing the standard GAN loss.
• The discriminator is trained to output a score that aligns with the actual SI-SNR value, enabling more stable and meaningful training signals.
• The generator is optimized via a loss function that encourages the enhanced speech to achieve high SI-SNR while being guided by the metric discriminator.
• The method is formalized as a Metric GAN, where the discriminator’s output is constrained to match the objective metric (SI-SNR), ensuring alignment with perceptual quality.
• Theoretical analysis shows that under certain conditions, WGAN is a special case of Metric GAN, explaining the performance advantage of the latter.

Experimental results

Research questions

• RQ1 Why does Metric GAN outperform WGAN in speech enhancement despite similar theoretical foundations?
• RQ2 Can metric evaluation in the discriminator stabilize training and reduce scale ambiguity caused by SI-SNR loss?
• RQ3 How does the choice of target metric value (q) affect the performance of the Metric GAN framework?
• RQ4 Under what conditions is Metric GAN equivalent to WGAN, and what explains the performance gap when they are not?

Key findings

• TSEGAN achieves better SI-SNR performance on the validation set than WGAN-based models, demonstrating improved objective metric performance.
• M-TSEGAN with a target value of q=20 outperforms M-TSEGAN with q=200, indicating that a smaller target value leads to better metric fitting and performance.
• The metric evaluation discriminator successfully mitigates the scale ambiguity issue caused by SI-SNR loss, leading to more stable training.
• Theoretical analysis confirms that WGAN is a special case of Metric GAN under specific conditions, providing a foundation for understanding the performance superiority of Metric GAN.
• The framework is flexible and allows integration with various TasNet variants, enabling broader application in time-domain speech enhancement.