[Paper Review] Deep Noise Suppression With Non-Intrusive PESQNet Supervision Enabling the Use of Real Training Data

Ziyi Xu, Maximilian Strake+1 more|arXiv (Cornell University)|Mar 31, 2021

Speech and Audio Processing34 references4 citations

TL;DR

This paper proposes a weakly supervised deep noise suppression framework that leverages a non-intrusive PESQNet to enable training with real noisy speech data without requiring clean references. By alternately training a complex mask-based FCRN and the PESQNet using both synthetic and real data, the method achieves a 0.09 MOS improvement over the DNS3 baseline and a 0.45 MOS gain in background noise quality, marking the first successful use of real data in non-GAN, reference-free speech enhancement training.

ABSTRACT

Data-driven speech enhancement employing deep neural networks (DNNs) can provide state-of-the-art performance even in the presence of non-stationary noise. During the training process, most of the speech enhancement neural networks are trained in a fully supervised way with losses requiring noisy speech to be synthesized by clean speech and additive noise. However, in a real implementation, only the noisy speech mixture is available, which leads to the question, how such data could be advantageously employed in training. In this work, we propose an end-to-end non-intrusive PESQNet DNN which estimates perceptual evaluation of speech quality (PESQ) scores, allowing a reference-free loss for real data. As a further novelty, we combine the PESQNet loss with denoising and dereverberation loss terms, and train a complex mask-based fully convolutional recurrent neural network (FCRN) in a "weakly" supervised way, each training cycle employing some synthetic data, some real data, and again synthetic data to keep the PESQNet up-to-date. In a subjective listening test, our proposed framework outperforms the Interspeech 2021 Deep Noise Suppression (DNS) Challenge baseline overall by 0.09 MOS points and in particular by 0.45 background noise MOS points.

Motivation & Objective

• To address the challenge of training deep speech enhancement models using real noisy speech data, which lacks clean reference signals.
• To develop a reference-free perceptual loss function that enables training with real data by estimating PESQ scores directly from enhanced signals.
• To overcome limitations of prior learned quality metrics (e.g., Quality-Net) that degrade when the enhancement model exploits them during training.
• To improve generalization and performance on real-world test data by combining synthetic and real data in a weakly supervised training protocol.
• To demonstrate that real data can be effectively used in non-GAN, perceptually driven training for speech enhancement.

Proposed method

• Proposes an end-to-end non-intrusive PESQNet that estimates PESQ scores from enhanced speech signals without access to clean references, enabling reference-free training.
• Introduces a weakly supervised training protocol (1-1-50) that alternates between updating the FCRN and the PESQNet, with each cycle using synthetic data, real data, and synthetic data again to keep the PESQNet up-to-date.
• Combines the PESQNet loss with a denoising and dereverberation loss from prior work (Jsynth_u), forming a total loss Jtotal_u that balances perceptual quality and signal fidelity.
• Uses a complex mask-based fully convolutional recurrent network (FCRN) for speech enhancement, trained with both synthetic data (for Jsynth_u) and real data (for PESQNet loss).
• Employs a novel training schedule where the PESQNet is retrained periodically using the latest FCRN outputs to prevent distribution shift and maintain accuracy.
• Leverages DNSMOS and subjective MOS (ITU-T P.835) to evaluate performance on real-world test sets, including the DNS3 blind test set.

Experimental results

Research questions

• RQ1 Can real noisy speech data be effectively used to train a deep speech enhancement model without access to clean reference signals?
• RQ2 Can a non-intrusive PESQNet provide a stable, differentiable, and perceptually meaningful loss for training with real data?
• RQ3 Does alternating training between the enhancement network and the PESQNet prevent performance degradation due to distribution shift or model exploitation?
• RQ4 Can the integration of real data via PESQNet loss improve performance on real-world test sets compared to fully synthetic training?
• RQ5 Is it possible to achieve state-of-the-art performance on real data without using generative adversarial networks (GANs)?

Key findings

• The proposed framework outperforms the DNS3 Challenge baseline by 0.09 MOS points in overall quality on the blind test set, demonstrating significant subjective improvement.
• The framework achieves a 0.45 MOS point improvement in background noise quality, indicating better preservation of naturalness and noise characteristics.
• On real data from the DNS3 development set, the model achieves a DNSMOS score of 3.33, outperforming all other methods except the oracle setup with full real data access.
• The 1-1-50 training protocol with α=0.9 enables effective use of real data, as shown by the 0.02-point DNSMOS gain over a synthetic-only baseline, despite limited real data (4 hours).
• The alternating training of FCRN and PESQNet successfully mitigates the degradation issue reported in prior work with fixed quality networks, maintaining PESQNet reliability over time.
• This work is the first to demonstrate effective, non-GAN-based training of speech enhancement models using real noisy data with a perceptual, reference-free loss.