[Paper Review] VSEGAN: Visual Speech Enhancement Generative Adversarial Network

Xinmeng Xu, Yan Wang+5 more|arXiv (Cornell University)|Feb 4, 2021

Speech and Audio Processing4 citations

TL;DR

This paper proposes VSEGAN, a generative adversarial network that enhances noisy speech using visual information from lip movements. By fusing audio and visual features through a multi-layer convolutional encoder-decoder architecture and training with a least-squares GAN loss, VSEGAN achieves state-of-the-art performance in speech quality and intelligibility, outperforming audio-only and baseline audio-visual models on the GRID dataset with a PESQ of 3.10 and STOI of 89.8% at 0 dB SNR.

ABSTRACT

Speech enhancement is an essential task of improving speech quality in noise scenario. Several state-of-the-art approaches have introduced visual information for speech enhancement,since the visual aspect of speech is essentially unaffected by acoustic environment. This paper proposes a novel frameworkthat involves visual information for speech enhancement, by in-corporating a Generative Adversarial Network (GAN). In par-ticular, the proposed visual speech enhancement GAN consistof two networks trained in adversarial manner, i) a generator that adopts multi-layer feature fusion convolution network to enhance input noisy speech, and ii) a discriminator that attemptsto minimize the discrepancy between the distributions of the clean speech signal and enhanced speech signal. Experiment re-sults demonstrated superior performance of the proposed modelagainst several state-of-the-art

Motivation & Objective

• To improve speech enhancement in noisy environments by leveraging visual information from lip movements.
• To address the limitations of audio-only speech enhancement models in preserving speech intelligibility and quality under low SNR conditions.
• To develop a robust, end-to-end audio-visual speech enhancement framework using deep generative modeling.
• To investigate whether adversarial training with visual cues improves the perceptual quality and intelligibility of enhanced speech.
• To demonstrate that multi-layer feature fusion and GAN-based training enhance the robustness and performance of visual speech enhancement.

Proposed method

• The model uses a conditional GAN framework where the generator takes noisy audio and visual video frames as input to produce enhanced speech.
• The generator employs a multi-layer feature fusion convolutional network with separate audio (Conv-A) and visual (Conv-V) encoders, followed by audio-visual fusion (Conv-AV) and a decoder with transposed convolutions.
• The discriminator is trained to distinguish between real clean speech and enhanced speech generated by the generator, using a least-squares GAN loss to mitigate vanishing gradients.
• The training objective combines a perceptual loss (L1) and a GAN loss, with a hyperparameter λ = 100 to balance the two components.
• Audio features are represented as log-Mel spectrograms (80 bands, 40ms window, 10ms hop), and video frames are processed as 5-frame clips with spatial and temporal convolutions.
• The model is trained for 70 epochs using ADAM optimizer with a fixed learning rate of 1e-4 and a batch size of 8.

Fig. 1 : Network architecture of generator. Conv-A, Conv-V, Conv-AV, BN, and Deconv denote convolution of audio encoder, convolution of video encoder, convolution of audio-visual fusion, batch normalization, and transposed convolution.

Experimental results

Research questions

• RQ1 Can visual information from lip movements significantly improve speech enhancement performance in noisy environments?
• RQ2 Does incorporating a GAN-based framework with adversarial training lead to better perceptual quality and intelligibility in audio-visual speech enhancement?
• RQ3 How does multi-layer feature fusion between audio and visual streams affect the performance and training stability of the enhancement model?
• RQ4 How does VSEGAN compare to state-of-the-art audio-visual speech enhancement models in terms of PESQ and STOI metrics?
• RQ5 Is the proposed model robust across varying SNR conditions, particularly at low SNR levels?

Key findings

• At 0 dB SNR, VSEGAN achieved a PESQ score of 3.10 and an STOI score of 89.8%, outperforming SEGAN (PESQ: 2.21, STOI: 77.3%) and the baseline model (PESQ: 2.94, STOI: 87.9%).
• At -5 dB SNR, VSEGAN achieved a PESQ of 2.88 and STOI of 86.8%, demonstrating strong performance even under severe noise conditions.
• On the GRID dataset, VSEGAN surpassed recent state-of-the-art models, including OVA (PESQ: 2.69, STOI: 89.75%) and AV(SE)2 (PESQ: 2.98, STOI: 89.44%), in both PESQ and STOI metrics.
• Visual feature fusion and GAN training significantly improved enhancement quality, as shown by the performance gap between SEGAN (audio-only) and VSEGAN (audio-visual GAN).
• Visualizations of spectrograms confirmed that VSEGAN preserves more fine spectral details and reduces noise more effectively than baseline and generator-only models.
• The model’s performance gain is attributed to both the integration of visual cues and the use of adversarial training, which enhances perceptual quality beyond what signal-based losses alone can achieve.

Fig. 2 : Network architecture of discriminator, and GAN training procedure.