[Paper Review] Speech denoising by parametric resynthesis

Soumi Maiti, Michael Mandel|arXiv (Cornell University)|Apr 2, 2019

Speech and Audio Processing18 references4 citations

TL;DR

This paper proposes a parametric resynthesis approach for speech denoising that predicts clean speech vocoder parameters from noisy input using a neural network, then resynthesizes high-quality, noise-free speech via a vocoder. The method achieves subjective quality and intelligibility on par with the oracle Wiener mask and surpasses DNN-based mask prediction, demonstrating that prosody from noisy speech enables superior speech quality compared to text-based synthesis.

ABSTRACT

This work proposes the use of clean speech vocoder parameters as the target for a neural network performing speech enhancement. These parameters have been designed for text-to-speech synthesis so that they both produce high-quality resyntheses and also are straightforward to model with neural networks, but have not been utilized in speech enhancement until now. In comparison to a matched text-to-speech system that is given the ground truth transcripts of the noisy speech, our model is able to produce more natural speech because it has access to the true prosody in the noisy speech. In comparison to two denoising systems, the oracle Wiener mask and a DNN-based mask predictor, our model equals the oracle Wiener mask in subjective quality and intelligibility and surpasses the realistic system. A vocoder-based upper bound shows that there is still room for improvement with this approach beyond the oracle Wiener mask. We test speaker-dependence with two speakers and show that a single model can be used for multiple speakers.

Motivation & Objective

• To improve speech denoising by leveraging prosody information present in noisy speech, which is lost in text-based TTS systems.
• To address the limitations of traditional speech enhancement, such as over-suppression of speech and under-suppression of noise.
• To explore whether clean speech vocoder parameters—designed for high-quality synthesis—can serve as effective targets for neural network-based denoising.
• To evaluate the performance of the proposed method against established baselines, including the oracle Wiener mask and DNN-based mask prediction.
• To investigate speaker independence and generalization of the model across multiple speakers using a single trained system.

Proposed method

• Train a neural network to predict clean speech acoustic parameters (spectral envelope, F0, voiced/unvoiced, aperiodic energy) from noisy log mel-spectrogram features.
• Use the WORLD vocoder to encode clean speech into acoustic parameters and decode predicted parameters back into waveforms, enabling high-fidelity resynthesis.
• Train the prediction model using mean squared error loss between predicted and ground-truth acoustic features, with features including first and second derivatives.
• Implement both feed-forward DNN and LSTM architectures to model temporal context, with the LSTM version achieving better performance.
• Apply the model to noisy speech by predicting clean parameters and resynthesizing the signal, producing a noise-free output.
• Use a vocoder-based upper bound to evaluate the potential ceiling of the method, showing that improvements beyond the oracle Wiener mask are still possible.

Experimental results

Research questions

• RQ1 Can clean speech vocoder parameters serve as effective targets for neural network-based speech denoising?
• RQ2 Does using prosody from noisy speech improve speech quality compared to text-based TTS systems?
• RQ3 How does the performance of parametric resynthesis compare to the oracle Wiener mask and DNN-based mask prediction in terms of subjective quality and intelligibility?
• RQ4 Can a single model generalize across multiple speakers without requiring speaker-specific adaptation?
• RQ5 What is the upper bound on performance for this parametric resynthesis approach, and how does it compare to existing methods?

Key findings

• The parametric resynthesis model achieves subjective speech quality and intelligibility comparable to the oracle Wiener mask, which has access to the clean signal.
• The model outperforms the DNN-based mask predictor in both subjective quality and intelligibility, with a PESQ score of 2.43 versus 2.26 and STOI of 0.87 versus 0.80.
• The model produces noise-free speech, as confirmed by subjective MUSHRA testing, where it achieved higher noise suppression quality than the oracle Wiener mask.
• The model surpasses a standard statistical TTS system in both objective metrics and subjective quality, with a PESQ of 2.43 compared to 1.33 and STOI of 0.87 versus 0.08.
• A single model trained on multiple speakers achieves high performance across speakers, demonstrating effective speaker independence.
• The vocoder-based upper bound shows that the model still has room for improvement beyond the oracle Wiener mask, indicating potential for further optimization.