[Paper Review] VoiceFixer: Toward General Speech Restoration with Neural Vocoder

Haohe Liu, Qiuqiang Kong+5 more|arXiv (Cornell University)|Sep 28, 2021

Speech Recognition and SynthesisComputer Science76 references25 citations

TL;DR

VoiceFixer introduces a two-stage general speech restoration (GSR) framework that jointly handles multiple distortions with a mel-spectrogram analysis stage and a neural vocoder synthesis stage, achieving better MOS than single-task SSR baselines across diverse distortions.

ABSTRACT

Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on single-task speech restoration (SSR), such as speech denoising or speech declipping. However, SSR systems only focus on one task and do not address the general speech restoration problem. In addition, previous SSR systems show limited performance in some speech restoration tasks such as speech super-resolution. To overcome those limitations, we propose a general speech restoration (GSR) task that attempts to remove multiple distortions simultaneously. Furthermore, we propose VoiceFixer, a generative framework to address the GSR task. VoiceFixer consists of an analysis stage and a synthesis stage to mimic the speech analysis and comprehension of the human auditory system. We employ a ResUNet to model the analysis stage and a neural vocoder to model the synthesis stage. We evaluate VoiceFixer with additive noise, room reverberation, low-resolution, and clipping distortions. Our baseline GSR model achieves a 0.499 higher mean opinion score (MOS) than the speech enhancement SSR model. VoiceFixer further surpasses the GSR baseline model on the MOS score by 0.256. Moreover, we observe that VoiceFixer generalizes well to severely degraded real speech recordings, indicating its potential in restoring old movies and historical speeches. The source code is available at https://github.com/haoheliu/voicefixer_main.

Motivation & Objective

• Motivate and define the general speech restoration (GSR) task to restore multiple distortions with a single model.
• Propose VoiceFixer, a two-stage framework mimicking human auditory processing for improved restoration quality.
• Show that separating analysis and synthesis stages via mel-spectrogram representation yields strong performance across distortions.

Proposed method

• Two-stage architecture: analysis maps distorted audio to a mel-spectrogram representation; synthesis uses a neural vocoder to generate waveform from mel input.
• Analysis stage modeled by ResUNet to restore mel spectrograms from mel-filtered inputs.
• Synthesis stage uses a non-autoregressive vocoder (TFGAN) trained with adversarial losses and multi-domain spectro-temporal losses.
• Training losses include MAE for mel restoration, and a combination of time-domain and frequency-domain losses for the vocoder.
• Discriminators include multi-resolution time, subband, and frequency discriminators to guide vocoder training.
• vocoder loss combines L_F (mel loss and multi-resolution spectrogram losses) and L_T (segment, energy, phase) with an adversarial component (L_D).

Experimental results

Research questions

• RQ1 Can a general speech restoration (GSR) model restore multiple distortions in one framework?
• RQ2 Does a two-stage VoiceFixer architecture outperform one-stage SSR baselines on MOS and perceptual metrics across diverse distortions?
• RQ3 How well does VoiceFixer handle low sampling rates and combine analysis-synthesis to maintain quality?
• RQ4 What is the impact of different analysis architectures (ResUNet vs DNN/BiGRU) on restoration quality?
• RQ5 How does a neural vocoder trained on large speech data contribute to restoration performance?

Key findings

• VoiceFixer (VF) with UNet-based analysis achieves the highest MOS and LSD among evaluated systems on ALL-GSR.
• VF-UNet improves MOS by 0.256 over the GSR baseline (GSR-UNet) on ALL-GSR.
• The VF-UNet MOS is only 0.11 below the Oracle-Mel upper bound, indicating strong analysis-stage performance.
• VF demonstrates strong performance on low-sampling-rate super-resolution tasks, outperforming several SSR models especially at 2–8 kHz upsampling to 44.1 kHz.
• GSR-UNet generally outperforms SSR baselines across the ALL-GSR set, with VoiceFixer further enhancing perceptual quality.
• Vocoding-based synthesis benefits from a neural vocoder trained on large speech datasets, providing prior knowledge and lower-dimensional inputs.