[Paper Review] VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration

Haohe Liu, Xubo Liu+6 more|arXiv (Cornell University)|Apr 12, 2022

Speech and Audio Processing32 references6 citations

TL;DR

VoiceFixer is a unified deep learning framework for high-fidelity speech restoration that jointly handles multiple distortions—such as noise, reverberation, clipping, and low-bandwidth degradation—by using a two-stage approach: mel spectrogram estimation followed by neural vocoder-based waveform synthesis. It achieves a 0.256 higher MOS score than baseline methods and matches near-oracle performance on perceptual quality, even restoring severely degraded historical recordings.

ABSTRACT

Speech restoration aims to remove distortions in speech signals. Prior methods mainly focus on a single type of distortion, such as speech denoising or dereverberation. However, speech signals can be degraded by several different distortions simultaneously in the real world. It is thus important to extend speech restoration models to deal with multiple distortions. In this paper, we introduce VoiceFixer, a unified framework for high-fidelity speech restoration. VoiceFixer restores speech from multiple distortions (e.g., noise, reverberation, and clipping) and can expand degraded speech (e.g., noisy speech) with a low bandwidth to 44.1 kHz full-bandwidth high-fidelity speech. We design VoiceFixer based on (1) an analysis stage that predicts intermediate-level features from the degraded speech, and (2) a synthesis stage that generates waveform using a neural vocoder. Both objective and subjective evaluations show that VoiceFixer is effective on severely degraded speech, such as real-world historical speech recordings. Samples of VoiceFixer are available at https://haoheliu.github.io/voicefixer.

Motivation & Objective

• To address the limitation of existing speech restoration models that focus on only one type of distortion, such as denoising or dereverberation, in isolation.
• To develop a unified framework capable of restoring speech degraded by multiple simultaneous distortions, including noise, reverberation, clipping, and low bandwidth.
• To achieve high-fidelity (44.1 kHz) speech restoration from severely degraded inputs, such as historical recordings, improving perceptual quality beyond low-fidelity or single-distortion methods.
• To reduce computational cost and artifact accumulation by replacing cascaded models with an end-to-end trainable, unified architecture.
• To release a pre-trained model and codebase to accelerate future research in multi-distortion speech restoration.

Proposed method

• The framework uses a two-stage design: an analysis stage that estimates mel spectrograms from degraded speech using a ResUNet-based network, and a synthesis stage that generates high-fidelity waveforms via a neural vocoder (TFGAN).
• Intermediate features are represented as low-dimensional mel spectrograms, which serve as a shared representation to unify the restoration of diverse distortions.
• Distortions are modeled as a composite function of sequential operations: additive noise, convolution with room impulse response (reverberation), and amplitude clipping within a fixed range.
• The analysis module is trained to predict clean mel spectrograms from distorted inputs, while the synthesis module is pre-trained on large-scale speech datasets to generate high-quality waveforms from mel features.
• The entire system is trained in a two-stage fashion: analysis and synthesis are optimized separately, leveraging pre-trained vocoder priors for better waveform generation.
• The framework supports both multi-distortion restoration and speech super-resolution, enabling expansion of low-bandwidth speech (1–22.05 kHz) to 44.1 kHz full-bandwidth.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively restore speech degraded by multiple simultaneous distortions, such as noise, reverberation, clipping, and low bandwidth?
• RQ2 Does using mel spectrograms as intermediate features improve the robustness and performance of multi-distortion speech restoration compared to direct waveform or spectrogram modeling?
• RQ3 Can a unified framework achieve high-fidelity (44.1 kHz) speech restoration with perceptual quality approaching that of clean speech, even from severely degraded inputs?
• RQ4 How does the performance of VoiceFixer compare to cascaded models or task-specific baselines in terms of objective metrics (PESQ, SSIM, LSD) and subjective MOS?
• RQ5 To what extent can a pre-trained neural vocoder generalize to restore high-fidelity waveforms from estimated mel spectrograms in a unified restoration pipeline?

Key findings

• VoiceFixer achieves a mean opinion score (MOS) of 3.62 on the HiFi-Res test set, which is 0.256 higher than the baseline UNet method and only 0.11 below the Oracle-Mel reference.
• On the VD-Test denoising benchmark, VoiceFixer achieves a PESQ-wb score of 2.43, outperforming SEGAN, WaveUNet, and the weakly labeled model (WL-Model), and matches the Target MOS of 3.69.
• In the declipping task, VoiceFixer achieves a MOS of 3.38 at both clipping levels (0.25 and 0.1), outperforming SSPADE by 0.04 and 1.25 points respectively, demonstrating superior perceptual quality.
• Despite lower scores on objective metrics like PESQ-wb and SSIM, VoiceFixer shows the strongest improvement in subjective MOS, indicating that perceptual quality gains are not always captured by standard objective measures.
• The framework successfully restores low-bandwidth speech (1–22.05 kHz) to 44.1 kHz full-bandwidth, achieving high-fidelity output with enhanced high-frequency energy, which contributes to better listener perception.
• The release of pre-trained models and code enables future research in multi-distortion speech restoration and supports extension to additional distortion types.