[Paper Review] Solving Audio Inverse Problems with a Diffusion Model

Eloi Moliner, Jaakko Lehtinen+1 more|arXiv (Cornell University)|Oct 27, 2022

Music and Audio Processing4 citations

TL;DR

This paper introduces CQT-Diff, a diffusion-based generative model that leverages the pitch-equivariant structure of the Constant-Q Transform (CQT) to solve diverse audio inverse problems—such as bandwidth extension, inpainting, and declipping—without retraining. By preconditioning the diffusion model with an invertible CQT, the method achieves state-of-the-art performance in bandwidth extension and competitive results in inpainting and declipping, demonstrating the first general-purpose, problem-agnostic framework for audio restoration using diffusion models.

ABSTRACT

This paper presents CQT-Diff, a data-driven generative audio model that can, once trained, be used for solving various different audio inverse problems in a problem-agnostic setting. CQT-Diff is a neural diffusion model with an architecture that is carefully constructed to exploit pitch-equivariant symmetries in music. This is achieved by preconditioning the model with an invertible Constant-Q Transform (CQT), whose logarithmically-spaced frequency axis represents pitch equivariance as translation equivariance. The proposed method is evaluated with objective and subjective metrics in three different and varied tasks: audio bandwidth extension, inpainting, and declipping. The results show that CQT-Diff outperforms the compared baselines and ablations in audio bandwidth extension and, without retraining, delivers competitive performance against modern baselines in audio inpainting and declipping. This work represents the first diffusion-based general framework for solving inverse problems in audio processing.

Motivation & Objective

• To develop a unified, problem-agnostic framework for solving various audio inverse problems using a single pre-trained generative model.
• To address the limitation of existing methods that are tailored to specific degradation types and require task-specific training.
• To explore whether diffusion models, preconditioned with an invertible time-frequency transform, can generalize across diverse audio restoration tasks.
• To evaluate the effectiveness of pitch-equivariant inductive biases—specifically via the Constant-Q Transform—in improving audio restoration performance.
• To demonstrate that a single generative model can achieve competitive or state-of-the-art results across multiple audio restoration tasks without task-specific fine-tuning.

Proposed method

• The method employs a score-based diffusion model trained in the time domain, with an invertible Constant-Q Transform (CQT) used as a preconditioning step to exploit pitch-equivariance.
• The CQT's logarithmically spaced frequency axis ensures that pitch shifts correspond to translations, enabling the use of convolutional layers that respect harmonic structure.
• The model uses a noise schedule and stochastic reverse SDE to sample from the learned score function, with the score approximated by a neural network $ s_{\theta}(\bm{x}_{\tau}, \sigma_{\tau}) $.
• Conditioning for inverse problems is achieved via a reconstruction guidance (RG) term and, optionally, a denoising classifier (DC), enabling flexible inference with partial observations.
• The model is trained on solo piano music and evaluated on three tasks: bandwidth extension, inpainting, and declipping, using both objective and subjective metrics.
• The framework supports both linear and nonlinear inverse problems by conditioning on known degradation models, such as low-pass filtering, signal gaps, or hard clipping.

Experimental results

Research questions

• RQ1 Can a single pre-trained diffusion model generalize across multiple audio inverse problems without retraining?
• RQ2 Does preconditioning a diffusion model with an invertible CQT improve performance on audio restoration tasks by exploiting pitch-equivariant symmetries?
• RQ3 How does the performance of a CQT-based diffusion model compare to task-specific baselines in bandwidth extension, inpainting, and declipping?
• RQ4 What role does reconstruction guidance play in maintaining temporal coherence during audio inpainting?
• RQ5 Can a generative prior from a diffusion model outperform discriminative methods in severe degradation scenarios, such as low signal-to-distortion ratio (SDR) in declipping?

Key findings

• CQT-Diff achieves state-of-the-art performance in audio bandwidth extension, outperforming all compared baselines in both objective and subjective evaluations.
• In audio inpainting, CQT-Diff achieves marginally higher MUSHRA scores than GACELA, demonstrating competitive performance despite not being specifically trained for the task.
• For declipping at 1 dB SDR, CQT-Diff achieves the best FAD (1.84), PEAQ (-3.53), and PEMO-Q (-3.36) scores, outperforming A-SPADE and SS-PEW in the most severe degradation case.
• At 10 dB SDR, CQT-Diff's performance is outperformed by SS-PEW due to limitations in the generative model's quality, though it remains comparable to A-SPADE.
• The use of reconstruction guidance (RG) is critical for inpainting, as it significantly improves coherence and plausibility, indicating that the model's architecture alone is insufficient for long-term dependencies.
• The results confirm that the CQT preconditioning provides a meaningful inductive bias, enabling the model to generalize across diverse inverse problems using a single, unified architecture.